NodeJS, OpenCV and Streaming Images Using Net Socket

Question

My end goal is to stream video from my laptop to a server. I'm trying to accomplish this by using NodeJs on the laptop & the server. I use the OpenCV library to capture the video on the laptop and save it to a jpg file. I then read the file and convert it to base64 so that I can transport it using the Net.socket module in Node. This is a continous process: capture, encode, and send.

Here is the server code for just transmitting one jpg file:

var cv = require('opencv');
var fs = require('fs');
var net = require('net');
var camera = new cv.VideoCapture(0);
var server = net.createServer();
server.listen('50007', '127.0.0.1');

server.on('connection', function(socket){
    camera.read(function(image){
        image.save('original.jpg');

        fs.readFile('original.jpg', 'base64', function(err, image){
            socket.write(image, 'base64', function(){
                socket.end();
            });
        });
    });
});

On the client I loop until the FIN is received from the server. Here is the client code:

var net = require('net');
var fs = require('fs');
var client = new net.Socket();
var buffer ='';
client.setEncoding('base64');

client.connect('50007', '127.0.0.1', function(){
    console.log('Connecting to server...');
});

client.on('data', function(data){
    buffer += data;
});

client.on('end', function(){
    var dataBuffer = new Buffer(buffer, 'base64');
    fs.writeFile('copy.jpg', dataBuffer, function(err){
        if(err){
            console.log(err);
        }
    });
});

The problem is that the entire image does not actually get sent. When I open the received file, copy.jpg, there is always a chunk missing at the bottom.

In the final version the goal is to send one jpg after another and delimit the end of each 'jpg' via a keyword such as 'EndOfFile'. I tried to do this by appending the keyword 'EndOfFile' to my base64 encoded image before sending but on the receiving end that really got screwed up.

Sample Advanced Server:

fs.readFile('original.jpg', 'base64', function(err, image){
    image += 'EndOfFile';
    socket.write(image, 'base64');
});

One the client side the loop would examine each chunk of data for the keyword and if it found it then whatever is in the buffer would be written to file and the buffer reset, ready for the next file.

Sample Advanced Client

client.on('data', function(data){
    if(data.indexOf('EndOfFile') > 0){
        buffer += data.substr(0, data.indexOf('EndOfLine'));
        var dataBuffer = new Buffer(buffer, 'base64');

        fs.writeFile('copy.jpg', dataBuffer, function(err){
            if(err){
                console.log(err);
            }
        });

        buffer = '';
    } else {
        buffer += data;
    }
});

I've gotten this to work in Python so I think my logic is correct but I'm not as confortable in NodeJS.

If someone could tell me if this is a sane way to do this and where may I have gone wrong.

Thanks in advance!

I wonder how fast it is for your streaming. Actually, I was trying to do the same by sending img base64 from C/C++ opencv via socket. I was able to get it to work, but the frame rate is damn slow and very lag. I believe it need some encoding to make it faster. — A-letubby
Yes this approach is very slow because it does not use any compression. I ended up using something called GStreamer to take care of the streaming for me. I suggest that you use a similar tool that will do some kind of compression on the fly. At any rate, this was a good exercise for programmings sake but I would never do things this way for a production or even a demo system. — Przemek Lach
In GStreamer, how can you show your streaming video on webpage? Now, I am looking at webrtc approach which is still new but not standardised yet. It can send video streaming easily by webcam but streaming for video file still in a progress. — A-letubby
So it was a combination of GStream and servlets. At the end of the GStreamer pipeline we had a TCP sink and that sink was sending data to a servlet which then forwarded it to a HTML5 video tag. You can replace the servlet with whatever you want. The main idea is that the GStreamer pipeline ended with a TCP sink and then you need to connect your video player to that TCP sink. — Przemek Lach
More specifically the element was called tcpserversink. There was also an element before the sink which did the encoding of the video. I don't recall the name of the element but I remember that we either encoded is as a multi-part jpeg or vp8. — Przemek Lach

Kornel Kornel · Accepted Answer · 2015-04-18T09:44:52

I suspect you're seeing end event while the last bit of data is still buffered.

Try waiting for the close event rather than the end event. I'm not sure about sockets, but in other Node APIs like spawn, the end event is fired early, before related streams are flushed, so there may still be buffered data waiting.

You could avoid managing this yourself by piping. Use fs.createWriteStream() and .pipe() the socket stream to the file.

NodeJS, OpenCV and Streaming Images Using Net Socket

1 Answers