RabbitMQ as Message Broker used by Spring Websocket dies under load

Question

I develop an application where we need to handle 160k concurrent users which are connected to the backend via a websocket connection.

We decided to use the spring websocket implementation and RabbitMQ as the message broker.

In our application every user needs to subscribe to its user queue /exchange/amq.direct/update as well as to another queue where also other users can potential subscribe to /topic/someUniqueName.

In our first performance test we did the naive approach where every user subscribes to two new queues.

When running the test RabbitMQ dies silently when around 800 users are connected at the same time, so around 1600 queues are active (See the graph of all RabbitMQ objects here).

I read though that you should be careful opening many connections to RabbitMQ.

Now I wonder if the approach that is anticipated by Spring Websocket with opening one queue per user is a conceptional problem for systems with high load or if there is another error in my system.

Answer 1

Limiting factors for RabbitMQ are usually:

• memory (can be checked in dashboard) that needs to grow with number of messages and number of queues (if you don't use lazy queues that go directly to disk).
• maximum number of file descriptors (at least 1 per connection) that often defaults to too low values on many distributions (ref: https://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2012-April/019615.html)
• CPU for routing the messages

Answer 2

I did find the issue. I actually misconfigured the RabbitMQ service and just gave it a 1024 file descriptor limit. Increasing it solved the issue.