WebRTC: Tracks are Empty in ontrack Event - Frontend and Backend Implementation

158 views Asked by At

I am working on a WebRTC application where I am trying to establish a video streaming connection between a client (React frontend) and a server (Node.js backend) using WebRTC. However, I am facing an issue where the ontrack event on the server sides is not receiving the expected tracks, and they appear to be empty.

I have followed various tutorials and examples to set up the WebRTC connection, including SDP offer-answer exchange and ICE candidate negotiation. I have checked the ICE candidate exchange and WebRTC negotiation logs, and they seem to be progressing as expected. However, when I attempt to access the tracks inside the ontrack event, they are showing up as empty.

I am using RTCPeerConnection to handle the peer connection and getDisplayMedia to capture the video stream on the client side. On the server side, I am using the wrtc library to handle WebRTC functionality. The ICE candidate negotiation and SDP exchange seem to work fine.

I suspect there might be an issue with how I'm handling the tracks and adding them to the connection, but I'm having trouble identifying the exact problem. frontend code:

  useEffect(() => {
    let stream;
    if (streaming) {
      //startDisplay();
      const init = async () => {
        await navigator.mediaDevices
          .getDisplayMedia({
            video: true,
            audio: true,
          })
          .then(async (some) => {
            localStream = some;
            console.log(localStream);
            const peerConnection = new RTCPeerConnection();
            localStream.getTracks().forEach((track) => {
              console.log(track);
              peerConnection.addTrack(track, localStream);
            });
            //peerConnection.addTrack(track, localStream);
            peerConnection.onicecandidate = (e) => {
              console.log("client onice " + JSON.stringify(e));
              //console.log("e candidate " + e.candidate);
              if (e.candidate) {
                socket.emit("webrtc_ice", e.candidate);
              }
            };
            const offer = await peerConnection.createOffer({
              offerToReceiveVideo: true,
              offerToReceiveAudio: true,
            });
            console.log("offer " + JSON.stringify(offer))
            //await peerConnection.setLocalDescription(new RTCSessionDescription(offer));
            await peerConnection.setLocalDescription(offer);
            //socket.emit("webrtc_offer", {
            //  type: "webrtc_offer",
            //  sdp: offer,
            //});

            console.log("offer above webrtc" + JSON.stringify(offer))
            socket.emit("webrtc_offer", offer);
            socket.on("webrtc_answer", async (ans) => {
              console.log("webRTC answer " + ans);
              console.log("webRTC answer " + JSON.stringify(ans));
              await peerConnection.setRemoteDescription(ans);
            });
            socket.on("webrtc_ice", async (event) => {
              console.log("Socket event callback: webrtc_ice_candidate");

              console.log("ice from server" + JSON.stringify(event));
              // ICE candidate configuration.
              //var candidate = new RTCIceCandidate({
              //  sdpMLineIndex: event.label,
              //  candidate: event.candidate,
              //});
              await peerConnection.addIceCandidate(event);
              console.log("peer connection add ice candidate" + peerConnection);
            });
          });
      };
      init();
    }
  }, [streaming]);

backend code:

        this.io.on("connection", (socket) => {
            console.log("Socket Connected");
            //socket.on("video", (msg) => {
            //  console.log("Socket Connected");
            //  if (msg) {
            //    ffmpeg.stdin.write(msg);
            //  }
            //  //);
            //});
            socket.on("webrtc_ice", (event) => {
                console.log("Socket event callback: webrtc_ice_candidate");
              console.log("webrtc" + JSON.stringify(event));
                console.log("webrtc ice server " + event.candidate);

                peerConnection.ontrack = (event: MediaStreamTrack) => {
                    console.log("hello" + JSON.stringify(event));
                    console.log("hello" + event);
                    //ffmpeg.stdin.write(track);
                };
                //ICE candidate configuration.

                var candidate = new wrtc.RTCIceCandidate({
                    sdpMLineIndex: event.label,
                    candidate: event.candidate,
                });
                peerConnection.addIceCandidate(event);
            });
            socket.on("webrtc_offer", async (data) => {
                //this.handlePeerConnection(data.sdp);

                //const ffmpeg = spawn("ffmpeg", ops);
              console.log("client webrtc offer " + JSON.stringify(data));
                peerConnection.onicecandidate = (e: any) => {
                    console.log("e candidate sending" + e.candidate);
                        this.io.emit("webrtc_ice", e.candidate);
                };
                await peerConnection.setRemoteDescription(data);
                const ans = await peerConnection.createAnswer();
                await peerConnection.setLocalDescription(
                    //new wrtc.RTCSessionDescription(ans),
                    ans,
                );
              console.log("ans" + JSON.stringify(ans));
                this.io.emit("webrtc_answer", ans);

            });
            socket.on("disconnect", () => {
                console.log("Socket Disconnected");
            });
        });

I attempted to set up a WebRTC video streaming connection between a React frontend and a Node.js backend. I followed tutorials and examples to exchange SDP offers and answers, as well as ICE candidates. My expectation was that the ontrack event would fire the server sides, providing access to the received video tracks.But it get fires, and the track is empty

However, when I accessed the tracks inside the ontrack event, they were empty. To troubleshoot, I examined ICE candidate negotiation and WebRTC event logs, and they indicated successful progress. I ensured the use of the getDisplayMedia function on the client side and the wrtc library on the server side.

Despite the logs suggesting proper negotiation, the ontrack event did not contain the expected tracks. I have reviewed my track handling and addition to the RTCPeerConnection instances, but I'm unable to identify the root cause of the empty tracks.

0

There are 0 answers