Socket.io and multiple Dyno's on Heroku Node.js app. WebSocket is closed before the connection is established

Question

I'm building an App deployed to Heroku which uses Websockets.

The websockets connection is working properly when I use only 1 dyno, but when I scale to >1, I get the following errors

POST http://****.herokuapp.com/socket.io/?EIO=2&transport=polling&t=1412600135378-1&sid=zQzJJ8oPo5p3yiwIAAAC 400 (Bad Request) socket.io-1.0.4.js:2

WebSocket connection to 'ws://****.herokuapp.com/socket.io/?EIO=2&transport=websocket&sid=zQzJJ8oPo5p3yiwIAAAC' failed: WebSocket is closed before the connection is established. socket.io-1.0.4.js:2

I am using the Redis adaptor to enable multiple web processes

var io = socket.listen(server);
var redisAdapter = require('socket.io-redis');
var redis = require('redis');

var pub = redis.createClient(18049, '[URI]', {auth_pass:"[PASS]"});
var sub = redis.createClient(18049, '[URI]', {detect_buffers: true, auth_pass:"[PASS]"} );

io.adapter( redisAdapter({pubClient: pub, subClient: sub}) );

This is working on localhost (which I am using foreman to run, as Heroku does, and I am launching 2 web processes, same as on Heroku).

Before I implemented the redis adaptor I got a web-sockets handshake error, so the adaptor has had some effect. Also it is working occasionally now, I assume when the sockets match the same web dyno.

I have also tried to enable sticky sessions, but then it never works.

var sticky = require('sticky-session');
sticky(1, server).listen(port, function (err) {
  if (err) {
    console.error(err);
    return process.exit(1);
  }
  console.log('Worker listening on %s', port);
});

Answer 1

I'm the Node.js Platform Owner at Heroku.

WebSockets works on Heroku out-of-the-box across multiple dynos; socket.io (and other realtime libs) use fallbacks to stateless processes like xhr polling that break without session affinity.

To scale up socket.io apps, first follow all the instructions from socket.io:

• http://socket.io/docs/using-multiple-nodes/

Then, enable session affinity on your app (this is a free feature):

• https://devcenter.heroku.com/articles/session-affinity

Answer 2

I spent a while trying to make socket.io work in multi-server architecture, first on Heroku and then on Openshift as many suggest.

The only way to make it work on both PAAS is disabling xhr-polling and setting transports: ['websocket'] on both client and server.

On Openshift, you must explicitly set the port of the server to 8000 (for ws – 8443 for wss on socket.io client initialization, using the *.rhcloud.com server, as explained in this post: http://tamas.io/deploying-a-node-jssocket-io-app-to-openshift/.

Polling strategy doesn't work on Heroku because it does not support sticky sessions (https://github.com/Automattic/engine.io/issues/261), and on Openshift it fails because of this issue: https://github.com/Automattic/engine.io/issues/279, that will hopefully be fixed soon.

So, the only solution I found so far, is disabling polling and use websocket transport only.

In order to do that, with socket.io > 1.0 server-side:

var app = express();
var server = require('http').createServer(app);

var socketio = require('socket.io')(server, {
  path: '/socket.io-client'
});
socketio.set('transports', ['websocket']);

client-side:

var ioSocket = io('<your-openshift-app>.rhcloud.com:8000' || '<your-heroku-app>.herokuapp.com', {
    path: '/socket.io-client'
    transports: ['websocket']
})

Hope this will help.

Answer 3

It could be you need to be running RedisStore:

var session = require('express-session');
var RedisStore = require('connect-redis')(session);

app.use(session({
    store: new RedisStore(options),
    secret: 'keyboard cat'
}));

per earlier q here: Multiple dynos on Heroku + socket.io broadcasts

Answer 4

I know this isn't a normal answer, but I've tried to get WebSockets working on Heroku for more than a week. After many long conversations with customer support I finally tried out OpenShift. Heroku WebSockets are in beta, but OpenShift WebSockets are stable. I got my code working on OpenShift in under an hour.

http://www.openshift.com

I am not affiliated with OpenShift in any way. I'm just a satisfied (non-paying) customer.

Answer 5

I was having huge problems with this. There were a number of issues failing simultaneously making it a huge nightmare. Make sure you do the following to scale socket.io on heroku:

1. if you're using clusters make sure you implement socketio-sticky-session or something similar

2. client's connect url should not be https://example.com/socket.io/?EIO=3&transport=polling but rather https://example.com/ notably I'm using https because heroku supports it

3. enable cors in socket.io

4. specify only websocket connections

For you and others it could be any one of these.

if you're having trouble setting up sticky-session clusters here's my working code

var http = require('http');
var cluster = require('cluster');
var numCPUs = require('os').cpus().length;
var sticky = require('socketio-sticky-session');
var redis = require('socket.io-redis');
var io;

if(cluster.isMaster){
  console.log('Inside Master');
  // create the worker processes
  for (var i = 0; i < numCPUs ; i++){
    cluster.fork();
  }
}
else {
  // The worker code to be run is written inside
  // the sticky().
}

sticky(function(){
  // This code runs inside the workers.
  // The sticky-session balances connection between workers based on their ip.
  // So all the requests from the same client would go to the same worker.
  // If multiple browser windows are opened in the same client, all would be
  // redirected to the same worker.
  io = require('socket.io')({transports:'websocket', 'origins' : '*:*'});
  var server = http.createServer(function(req,res){
    res.end('socket.io');
  })


  io.listen(server);
  // The Redis server can also be used to store the socket state
  //io.adapter(redis({host:'localhost', port:6379}));

  console.log('Worker: '+cluster.worker.id);
    // when multiple workers are spawned, the client
    // cannot connect to the cloudlet.

    StartConnect(); //this function connects my mongodb, then calls a function with io.on('connection', ..... socket.on('message'...... in relation to the io variable above

    return server;
}).listen(process.env.PORT || 4567, function(){
  console.log('Socket.io server is up ');
});

more information: personally it would work flawlessly from a session not using websockets (I'm using socket.io for a unity game. It worked flawlessly from the editor only!). When connecting through the browser whether chrome or firefox it would show these handshaking errors, along with error 503 and 400.