Using Redis pub/sub creates too many threads

Question

I have a Rails front-end server, which receives multiple requests from users, then send these requests to backend server.

Backend server processes requests asynchronously and notifies front-end server when it finishes each of the requests.

I use Redis pub/sub to communicate between these two servers. In particular, for each request coming from users, I create a new Redis instance that subscribes to the single channel (say, scoring_channel).

However, if I have 100 users making requests at the same time, each of the Redis subscribers will hold one thread.

Does this affect my server performance? If I have a constraint on maximum number of threads (e.g., Heroku allows max 256 threads), how should I avoid this issue?

If all the threads are subscribing to same channel then isn't so many subscriptions are too much cost?I feel subscription if not unique should be common. Also if each thread subscribes to same channel then all threads would get the same data, is this required as part of your design? — Nik

yinqiwen yinqiwen · Accepted Answer · 2014-02-24T05:03:14

This would not affect server performance since redis never blocked by pub/sub. You should use non-blocking API in client side instead of blocking version to decrease number of threads.

Using Redis pub/sub creates too many threads

1 Answers