Processing video data captured with WebRTC (Socket.IO) in a C++ server

Question

I'm trying to figure out the best way of doing the following things:

(A) Capture video frames in a client PC using a web browser (e.g. Chrome / Chromium).

(B) Send them to a server machine running a C++ processing algorithm (e.g. an OpenCV-based mutant squirrel detection routine).

(C) Send processing results back to the browser in the client PC in order to show them (real time would be nice, but overall latency is of course expected).

What I have discovered so far is that WebRTC is great for doing (A) (getUserMedia(), etc..) and, in theory, (B). My problems arise when it comes to send data to a C++ server. I have been looking to the basics of WebRTC (including the examples for P2P communication and the Native C++ API documentation) but I still have no clues on how to start building my server and send the data from the browser. Although I have little experience in Javascript, I have already worked in similar scenarios (Javascript Client <--> C++ Server with webSockets), but I though the webRTC solution should be even easier to implement.

Am I right about using webRTC for this scenario? Am I missing something? Is there any tutorial or example covering my scenario that I have missed?

Answer 1

From your post, I understand you need a WebRTC C Client to stablish a RTCPeerConnection between the C-client and the browser. After that, you can process the stream and return it to the browser using another RTCPeerConnection.

I don't work usually with C/C++ but check this repository: https://github.com/mozilla/webrtc-standalone , perhaps it can be useful.

Answer 2

So all you want to do the video sharing by using Webrtc, web sockets and c++ server.

We need the signaling server(in your case it's c++) for establish the connection between browsers, After established the connection the browsers share the video without server.

You can use the libwebsocket for establish the connection between browsers.

There is need to share the various data of user's browser, computer like audio video information, ip address etc.

You need to create peerConnection object by new RTCPeerConnection(configuration); on each browser and need to share the various information by creatOffer() and createAnswer() methods.

After shared relative information there will be established the connection between browsers and your video would start to share.

What is the process?(giving some summerize)

On your client side, how you can send/get request to/from c++ server.

1) Create socket on browser1 and browser2

var websocket = new WebSocket('ws://127.0.0.1:9000', 'echo');

2) Browser 1 (creat offer for browser 2)

websocket.send("createOfferForVideo");

3) Server

After got the request, server does broadcast the response to browser2.

printf("received data: %s\n", (char *) in);

4) Browser 2 (create answer for browser1 after got the offer)

websocket.onmessage = function (message){
   if(messageAboutCreateOffer){
       websocket.send("createAnswerForVideo");
   }
}

That's how the establish the connection between browsers and video would start to share.

For more information about WEBRTC

Library libwebsocket with c++ and websockets.

Useful examples are Example1, Example2

Answer 3

RTCPeer connection of webRTC is used to establishing connection between client to client without a server in between to transfer data . For this connection between clients we need servers for NAT traversal (Stun and Turn servers) , But once the connection is made data can be send bidirectional from client to client without storing in server.

If you want to send data to server and process it websockets must be used.

Answer 4

See if you really need to use c++ to process data (unlikely) as this creates big overhead if you even can easily implement it and undermines peer2peer advantages e.g. doesn;t burden your servers.

Process data in javascript and do facial recognition (and much more) on client side and there are many libraries that do it. Look at webrtc.io modules and they have rtc-videoproc ect for processing data.

can use plugin to use c++ in browser ( slow to load ).