ZeroMQ, Redis and Gevent

Question

I'm developing a server which has to process a big amount of small data requests.
The server is developed with ZeroMQ, using the PULL/PUSH pattern.
Basically each request process consists in storing its data (message) in Redis.

I would like to have better performance by processing each request with gevent.
The following code doesn't work (messages are not stored in Redis).

from gevent import monkey
monkey.patch_all()
from redis import Redis, StrictRedis
from redis import connection
import zmq
import gevent


context = zmq.Context()
socket = context.socket(zmq.PULL)
socket.bind("tcp://*:5000")

connection.socket = gevent.socket
redis = Redis()

def enqueue(message):
    redis.lpush('work_queue', message)


while True:
    message = socket.recv()
    #print message
    gevent.spawn(enqueue, message)

If I remove gevent stuff, then the code works (messages are stored correctly).

P.S. I have just started to play with zeromq and gevent.

Update: I wanted to implement a simple task queue. In ended up using Celery. The server (which use zmq) asynchronously starts celery tasks (and it works great).

Answer 1

To use zeromq with gevent, you have to import de zmq.green module.

import zmq.green as zmq

http://zeromq.github.io/pyzmq/api/zmq.green.html

Answer 2

You don't appear to be starting the event loop. According to the tutorial at http://sdiehl.github.io/gevent-tutorial/ try wrapping the spawn call in a gevent.joinall call. Try that to see if it works then. Caution, that may not be the ideal way to use gevent. But it should be a start.

Answer 3

with redis-py pyzmq, i write a demo, gevent version 1.0.1,

References:
(sorry, my reputation is less than 10, i could't post more than two 2 links):

[ gevent with redis-py ]
[ gevent with zmq ]

the reference links just show in the demo code :)

the demo works well. [ My demo code ]