Distributing task with python and zeromq

Question

I have a working application using python and zeromq and I would like to optimize it.

Briefly, a master node send the same request to all workers (about 200) and the then collect the answers. Based on the answer, it sends a message back to one node and the node answers back.

Right now I implemented a very simple pattern. Each worker has one REP socket and the server has a list of REQ sockets. The server iterates through all sockets sending the general message and then iterates through all sockets to collect the answers. Finally, based on the answers the server picks one worker, sends a message to it and waits for the reply.

This is, of course, quite slow. The slowest part is sending 200 times the same message. Collecting is also slow. The solutions that I have found to distribute tasks and collect answers do load balance which is not what I need. I need that each worker receives the message and responds.

What is the pattern recommended for this situation?

Thanks

What do you mean "What is the pattern recommended for this situation?" — Marcin
For example, if you have to process independent tasks, you can use the ventilator pattern to distribute the task as described here taotetek.wordpress.com/2011/02/02/… — Hernan
I know what a pattern is. You haven't actually explained what you are doing, what parts of your current solution are required, or even why distributing your messages is slow. The classic solution for "distributing things is slow" is to not distribute things, or do less of it. — Marcin
Thanks for trying to help me think the problem again but please give some credit. I need to distribute because the workers need to access information which does not fit in the RAM of one node (and accessing the disk is to slow). Distributing the messagetakes two orders of magnitude longer that the round trip to each node. That makes sense as I am doing it 200 times. That round trip includes all the overhead of going through the network stack. I was wondering if there was a pattern batches all this calls. — Hernan

mgoetzke mgoetzke · Accepted Answer · 2012-03-25T06:41:54

If each worker should need a different job, Pub/sub wont work . What you need then is a worker pool Implementation. Either you push out the jobs in a round robin style (just use the push socket bound on the server and have each client pull from it, zeromq will do the round robin) or you have each worker request a job from the server, if the jobs are at least of some minimum complexity and the variance between jobs is high that is a better approach. There are numerous examples in the zeromq guide on its homepage and on the net:

http://zguide.zeromq.org/page:all (look at the paronoid pirate pattern)
http://blog.garambrogne.net/post/2010/10/23/simple-python-work-queue-with-zeromq
https://github.com/marklit/zeromq-worker-queue

The concrete implementation also depends on whether or not you need to have jobs reliably processed.

Distributing task with python and zeromq

2 Answers