Load Balance: Node.js - Socket.io - Redis

Question

I have 3 Servers running NodeJs, and they are related each other with Redis (1 master, 2 slaves).

The issue i'm having is that running the system on a single server works fine, but when I scale it to 3 NodeJS servers, it starts missing messages and the system gets unstable.

My load balancer does not accept sticky sessions. So every time that the requests from the client arrives to it, they can go to a different server.

I'm pointing all the NodeJS servers to the Redis Master.

It looks like socket.io is storing information on each server and it is not being distributed with redis.

I'm using socket.io V9, I'm suspecting that I don't have any handshake code, could this be the reason?

My code to configure socket.io is:

var express = require('express');
var io = require('socket.io');
var redis = require('socket.io/node_modules/redis');
var RedisStore = require('socket.io/lib/stores/redis');

var pub = redis.createClient("a port", "an ip");
var sub = redis.createClient("a port", "an ip");
var client = redis.createClient("a port", "an ip");
var events = require('./modules/eventHandler');

exports.createServer = function createServer() {
    var app = express();
    var server = app.listen(80);
    var socketIO = io.listen(server);

    socketIO.configure(function () {
        socketIO.set('store', new RedisStore({
            redisPub: pub,
            redisSub: sub,
            redisClient: client
        }));
        socketIO.set('resource', '/chat/socket.io');
        socketIO.set('log level', 0);
        socketIO.set('transports', [, 'htmlfile', 'xhr-polling', 'jsonp-polling']);
    });

    // attach event handlers
    events.attachHandlers(socketIO);

    // return server instance
    return server;
};

Another way i'm thinking to do it (in the case that socket.io does not work without sticky sessions) is to catch every request to socket io before they are executed and use pub-sub with redis so they arrive to the correct node.js instance. any thoughts? — Ricardo Daniel
I'm checking this: github.com/nodejitsu/node-http-proxy so i can check first the message that arrive and the route them to the correct server — Ricardo Daniel

Eli Eli · Accepted Answer · 2013-06-10T20:09:49

Redis only syncs from the master to the slaves. It never syncs from the slaves to the master. So, if you're writing to all 3 of your machines, then the only messages that will wind up synced across all three servers will be the ones hitting the master. This is why it looks like you're missing messages.

More info here.

Read only slave

Since Redis 2.6 slaves support a read-only mode that is enabled by default. This behavior is controlled by the slave-read-only option in the redis.conf file, and can be enabled and disabled at runtime using CONFIG SET.

Read only slaves will reject all the write commands, so that it is not possible to write to a slave because of a mistake. This does not mean that the feature is conceived to expose a slave instance to the internet or more generally to a network where untrusted clients exist, because administrative commands like DEBUG or CONFIG are still enabled. However security of read-only instances can be improved disabling commands in redis.conf using the rename-command directive.

You may wonder why it is possible to revert the default and have slave instances that can be target of write operations. The reason is that while this writes will be discarded if the slave and the master will resynchronize, or if the slave is restarted, often there is ephemeral data that is unimportant that can be stored into slaves. For instance clients may take information about reachability of master in the slave instance to coordinate a fail over strategy.

Load Balance: Node.js - Socket.io - Redis

2 Answers