Some problems when scaling Socket.IO to multiple Node.js processes using cluster

Question

My node.js server uses cluster module in order to work on multiple processes.

If the server receives requests from clients with Socket.IO, it conveys the data to another server with redis publish. And it receive refined data with redis subscribe, and then it just toss this data to clients.

I use one node process to receive data with redis sub, and other processes to send data to clients with socket.io.

And the client connect socket.io when page loaded. Here, this is my problem. The connect event occured repeatedly not even the page loaded. When the client connect, I get the socket.id from that socket, and I use it later when I want to send data to that client socket. But this connect occur repeatedly, I think socket that client use changed. So, the first socket.id that I remembered will be useless. I can't send data from that socket.id. I stored auth information in the socket object, so the changed client socket is no help.

index.pug

    $(document).ready(function(){
       var socket = io.connect();
       (...)

app.js

    var cluster = require('cluster');
    var socketio = require('socket.io');
    var NRP = require('node-redis-pubsub');
    var nrpForChat = new NRP(config.chatRedisConfig);
    var nrpForCluster = new NRP(config.clusterRedisConfig);



    var startExpressServer = function(){
      var app = express();
      var server = require('http').createServer(app);
      var io = socketio.listen(server);
      var redis = require('socket.io-redis');
      io.adapter(redis({ host: 'localhost', port: 6380 }));


      io.sockets.on('connection', function(socket){
        socketController.onConnect(io, socket, nrpForChat);
      });


      server.listen(config.port, function(){
        console.log('Server app listening on port '+config.port);
      });


      nrpForCluster.on('to:others:proc', function(data){
        var socket = io.sockets.connected[data.target.sockid];
        if (socket) {
          if (data.event == '_net_auth') {
            if (data.data.res){
              socket.enterId = data.data.data.enterId;
              socket.memberKey = data.data.data.memberKey;
              socket.sid = data.data.data.sid;
              socket.emit(data.event, data.data);
            }else{
              console.log('auth failed.');
            }
          }     
        } else {
          socket.emit(data.event, data.data);
        }
      });

      module.exports = app;
    }


    var numCpus = require('os').cpus().length;

    if (cluster.isMaster) {
      for (var i = 0; i < numCpus; i++) {
        cluster.fork();
      }
    }
    else {
      if (cluster.worker.id == numCpus) {

        nrpForChat.on('chat:to:relay', function(data){
          nrpForCluster.emit('to:others:proc', data);
        });

        if (numCpus == 1) {
          startExpressServer();
        }
      }
      else {
        startExpressServer();
      }
    }

jfriend00 jfriend00 · Accepted Answer · 2017-03-10T17:50:26

By default, socket.io connects with several consecutive http requests. It essentially starts in HTTP polling mode and then after some initial data exchange, it switches to a webSocket transport.

Because of this, a cluster that does not have any sort of sticky load balancing will not work. Each of the initial consecutive http requests that are all supposed to go to the same server process will probably be sent to different server processes in the cluster and the initial connection will not work.

There are two solutions that I know of:

Implement some sort of sticky load balancing (in the clustering module) so that each client repeatedly goes to the same server process and thus all the consecutive http requests at the beginning of a connection will go to the same server process.
Switch your client configurations to immediately switch to the webSocket transport and never use the HTTP polling. The connection will still start with an http request (since that's how all webSocket connections start), but that exact same connection will be upgraded to webSocket so there will only ever be one connection.

FYI, you will also need to make sure that the reconnect logic in socket.io is properly reconnecting to the original server process that is was connected to.

socket.io has node.js clustering support in combination with redis. While the socket.io documentation site has been down for multiple days now, you can find some info here and Scaling Socket.IO to multiple Node.js processes using cluster and here's a previously cached version of the socket.io doc for clustering.

Some problems when scaling Socket.IO to multiple Node.js processes using cluster

1 Answers