Can multiple sidekiq instances process the same queue

23
votes

I'm not familiar with the internals of Sidekiq and am wondering if it's okay to launch several Sidekiq instances with the same configuration (processing the same queues).

Is it possible that 2 or more Sidekiq instances will process the same message from a queue?

UPDATE:
I need to know if there is a possible conflict, when running Sidekiq on more than 1 machine

2
rubysidekiq
Why would you want to do that? The advantage of sidekiq over resque is supposed to be multi-threading, so only one process is needed. - Thilo
I need to know if there is a possible conflict, when running Sidekiq on more than 1 machine. Its not a question about threading vs. processes. - Jacob
@Thilo, even on a single machine, while "only one process is needed" that limits the cpu use to 100% of a single core, which limits the speed of processing the queue. I have an 8-core machine and want siekiq to use to use 4 cores at full capacity, so I need 4 processes. - Ben Lee

2 Answers

46
votes

Yes, sidekiq can absolutely run many processes against the same queue. Redis will just give the message to a random process.

10
votes

Nope, I've ran Sidekiqs in different machines with no issues.

Each of the Sidekiqs read from the same redis server, and redis is very robust in multi-threaded, and distributed scenarios.

In addition, if you look at the web interface for Sidekiq, it will show all the workers across all machines because all the workers are logged in the same redis server.

So no, no issues.