Lots of socket errors with celery eventlet tasks

Question

I'm getting a lot of "IOError: Socket closed" exceptions from amqplib.client_0_8.method_framing.read_method when running my celery workers with the --pool=eventlet option. I'm also seeing a lot of timeout exceptions from eventlet.hubs.hub.switch.

I'm using an async_manage.py script similar to the one at https://gist.github.com/821848, running the works like:

./async_manage.py celeryd_detach -E --pool=eventlet --concurrency=120 --logfile=<path>

Is this a known issue, or is there something wrong with my configuration or setup?

I'm running djcelery 2.2.4, Django 1.3, and eventlet 0.9.15.

I'm monkey patching every, but I'm not 100% certain that nothing is blocking. What's the best way to find out, and what can I do if they are? — Ben Dowling
monkey patching only patches what eventlet knows how to patch, so you could still use libraries that are not covered by that. See here for example: unethicalblogger.com/2010/08/28/… — asksol
I've enabled blocking detection and it occasionally blocks at "return self.connection.commit()" in django/db/backends/__init__.py - I'm using MySQL. Is there any way to avoid this? — Ben Dowling
Discovered the undocumented MySQLdb option to monkey_patch - doing more testing now! — Ben Dowling

Ben Dowling Ben Dowling · Accepted Answer · 2011-06-22T11:43:50

The problem was a side effect of some code that was blocking. I managed to detect the blocking code using the eventlet option described in this article.

There were 2 places where blocking was occuring: DNS lookups, and MySQL database access. I managed to resolve the first by installing the dnspython package, and the second my using the undocumented MySQLdb option in eventlet:

import eventlet
eventlet.monkey_patch()
eventlet.monkey_patch(MySQLdb=True)

Lots of socket errors with celery eventlet tasks

1 Answers