Celery task is hanging with http request

Question

I'm testing celery tasks and have stumbled on issue. If in task exists code with request(through urllib.urlopen) then it's hanging. What reasons can be?

I just try start on minimal config with Flask. I used rabbitmq and redis for broker and backend, but result is the same.

file(run_celery.py) with tasks:

...import celery and flask app...

celery = Celery(
    app.import_name,
    backend=app.config['CELERY_BROKER_URL'],
    broker=app.config['CELERY_BROKER_URL']
)

@celery.task
def test_task(a):
    print(a)
    print(requests.get('http://google.com'))

In this way I launched worker: celery -A run_celery.celery worker -l debug

After this, I run ipython and call task.

from run_celery import test_task
test_task.apply_async(('sfas',))

Worker's beginning perform task:

...
Received task: run_celery.test_task...
sfas
Starting new HTTP connection (1)...

And after this it's hanging.

This behavior is actual only if task contain request. What Did I do wrong?

What is the status of your task in redis? What happens if you add a timeout to the request? — bwarren2
@bwarren2 Nothing. Python must to raise exception, if I set timeout for urlopen, but nothing happened. It looks like celery task is hanging. — Greg Eremeev
@bwarren2 When task is performing and hanging, within redis exists only 3 keys: 127.0.0.1:6379> keys * 1) "_kombu.binding.celeryev" 2) "_kombu.binding.celery.pidbox" 3) "_kombu.binding.celery" — Greg Eremeev
A few more things to try: what happens if you set a (short) task time limit? What about BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 10} # 10s. ? It is not clear to me where the problem is yet, so more tweaking might be informative. — bwarren2
@bwarren2 Thanks for your time. I found reason, but need to it more explore. — Greg Eremeev

Greg Eremeev Greg Eremeev · Accepted Answer · 2015-06-09T12:13:40

I found reason in my code and very wondered O_o. I don't know why this is happening but within file with tasks, exists import Model and when it is executing then perform initialization instance MagentoAPI(https://github.com/bernieke/python-magento). If I comment out this initialization then requests in celery tasks perform correctly.

Celery task is hanging with http request

1 Answers