Node.js listen to session variable change and trigger server-sent event

Question

I am writing a webapp, using express.js.

My webapp achieves the following

1. User posts 100 json objects
2. Each json object is processed via a service call
3. Once the service call is completed, a session variable is incremented
4. On incrementation of the session variable, a server side event must be sent to the client to update the progress bar

How do i achieve listening on a session variable change to trigger a server-sent event?

Listening to a variable change is not the only solution I seek?

I need to achieve sending a server-sent event once a JSON object is processed.

Any appropriate suggestion is welcome

Edit (based on Alberto Zaccagni's comment)

My code looks like this:

function processRecords(cmRecords,requestObject,responseObject)
{
    for (var index = 0; index < cmRecords.length; index++) 
    {
        post_options.body = cmRecords[index];
        request.post(post_options,function(err,res,body)
        {
            if(requestObject.session.processedcount)
                requestObject.session.processedcount = requestObject.session.processedcount + 1;
            else
                requestObject.session.processedcount = 1;

            if(err)
            {
                appLog.error('Error Occured %j',err);
            }
            else
            {
                appLog.debug('CMResponse: %j',body);
            }

            var percentage = (requestObject.session.processedcount / requestObject.session.totalCount) * 100;

            responseObject.set('Content-Type','text/event-stream');
            responseObject.json({'event':'progress','data':percentage});
        });
    };


}

1. When the first record is updated and a server side event is triggered using the responseObject (express response object)

2. When the second record is updated and I try triggering a server side event using the same responseObject. I get an error saying cannot set header to a response that has already been sent

Answer 1

It's hard to know exactly what the situation is without seeing the routes/actions you have in your main application...

However, I believe the issue you are running into is that you are trying to send two sets of headers to the client (browser), which is not allowed. The reason this is not allowed is because the browser does not allow you to change the content type of a response after you have sent the initial response...as it uses that as an indicator of how to process the response you are sending it. You can't change either of these (or any other headers) after you have sent them to a client once (one request -> one response -> one set of headers back to the client). This prevents your server from appearing schizophrenic (by switching from a "200 Ok" response to a "400 Bad Request," for example).

In this case, on the initial request, you are telling the client "Hey, this was a valid request and here is my response (via the status of 200 which is either set elsewhere or being assumed by ExpressJS), and please keep the communication channel open so I can send you updates (by setting your content type to text/event-stream)".

As far as how to "fix" this, there are many options. When I've done this, I've used the pub/sub feature of redis to act as the "pipe" that connects everything up. So, the flow has been like this:

1. Some client sends a request to /your-event-stream-url

   In this request, you set up your Redis subscriber. Anything that comes in on this subscription can be handled however you want. In your case, you want to "send some data down the pipe to the client in a JSON object with at least a data attribute." After you have set up this client, you just return a response of "200 Ok" and set the content type to "text/event-stream." Redis will take care of the rest.

2. Then, another request is made to another URL endpoint which accomplishes the task of "posting a JSON object" by hitting /your-endpoint-that-processes-json. (Note: obviously this request may be made by the same user/browser...but the application doesn't know/care about that)

   In this action, you do the processing of their JSON data, increment your counters, or do whatever...and return a 200 response. However, one of the things you'd do in this action is "publish" a message on the Redis channel your subscribers from step #1 are listening to so the clients get the updates. Technically, this action does not need to return anything to the client, assuming the user will have some type of feedback based on the 200-status code or on the server-sent event that is sent down the pipe...

A tangible example I can give you is this gist, which is part of this article. Note that the article is a couple years old at this point so some of the code may have to be tweaked a bit. Also note this is not guaranteed to be anything more than an example (ie: it has not been "load tested" or anything like that). However, it may help you get started.

Answer 2

I came up with a solution please let me know if this is the right way to do stuff ?

Will this solution work across sessions ?

Server side Code

var events = require('events');
var progressEmitter = new events.EventEmitter();

exports.cleanseMatch = function(req, res)
{
    console.log('cleanseMatch Inovked');
    var progressTrigger = new events.EventEmitter;
    var id = '';
    var i = 1;  

     id = setInterval(function(){
        req.session.percentage = (i/10)*100;
        i++;
        console.log('PCT is: ' + req.session.percentage);
        progressEmitter.emit('progress',req.session.percentage)

        if(i == 11) {
        req.session.percentage = 100;
        clearInterval(id);              
        res.json({'data':'test'});
        }
    },1000);
}

exports.progress = function(req,res)
{
    console.log('progress Inovked');
    // console.log('PCT is: ' + req.session.percentage);

    res.writeHead(200, {'Content-Type': 'text/event-stream'});
    progressEmitter.on('progress',function(percentage){
         console.log('progress event fired for : ' + percentage);
         res.write("event: progress\n");
         res.write("data: "+percentage+"\n\n");
    });
}

Client Side Code

var source = new EventSource('progress');
source.addEventListener('progress', function(e) {
    var percentage = JSON.parse(e.data);
    //update progress bar in client
    App.updateProgressBar(percentage);
}, false);