Using RabbitMQ with workers that run multithreaded jobs - Parallel Gem

Question

I am building a system in ruby (rabbitmq, parallel gem) that takes a list of jobs, queues them up in rabbit and then has workers pop jobs off the queue to execute them.

It is easy to get a worker to pop a single job off of the queue when it is ready but I would like to have each worker run 5 threads such that when all 5 threads are processing, that worker does not pop any jobs off the queue. When a thread becomes free, the worker accepts a job from the queue.

Using the Parallel gem, the only way I see to create multi-threaded processes is with the following code.

results = Parallel.map(array, :in_processes => MAX_PROCESSES) do |item|
 item.process
end

I would like to do something like

while true
 cur_threads = Parallel.get_cur_threads
 if cur_threads < MAX_PROCESSES
  # get another job from queue
  # allocate a thread for the job
end

Any ideas??

Micharch54 Micharch54 · Accepted Answer · 2012-07-30T16:27:02

I know this is really old but I think you'd probably get what you're looking for using the work_queue gem.

Using RabbitMQ with workers that run multithreaded jobs - Parallel Gem

1 Answers