Scheduled Celery tasks on RabbitMQ remain unacknowledged past their specified run time

Question

Having trouble getting scheduled tasks to run at a specified future time while using Celery and RabbitMQ.

Using Django on a Heroku server, with the RabbitMQ add-on.

The Problem:

Sometimes the tasks don't run at all, sometimes they do run, but the times that they run at are off by a significant margin (like an hour).

Example task that did not run:

When I try to run a task with a countdown or ETA, it never actually executes. This is an example ETA task that did not run:

>>> dummy_task.apply_async(eta=datetime.datetime.now() + timedelta(seconds=60))
<AsyncResult: 03001c1c-329e-46a3-8180-b115688e1865>

Resulting Log:

2012-07-24T14:03:08+00:00 app[scheduler.1]: [2012-07-24 10:03:08,909: INFO/MainProcess]
    Got task from broker: events.tasks.dummy_task[910ff406-d51c-4c29-bdd1-fec1a8168c12]     
    eta:[2012-07-24 10:04:08.819528+00:00]

One minute later nothing happens. The unacknowledged message count in my Heroku RabbitMQ management console increases by one and stays there.

This works:

I've made sure that the celery task is properly registered and RabbitMQ is configured to accept tasks by verifying that I can run the task using the delay() method.

>>> dummy_task.delay()
<AsyncResult: 1285ff04-bccc-46d9-9801-8bc9746abd1c>

Resulting Log:

2012-07-24T14:29:26+00:00 app[worker.1]: [2012-07-24 10:29:26,513: INFO/MainProcess] 
    Got task from broker: events.tasks.dummy_task[1285ff04-bccc-46d9-9801-8bc9746abd1c]

....

2012-07-24T14:29:26+00:00 app[worker.1]: [2012-07-24 10:29:26,571: INFO/MainProcess] 
    Task events.tasks.dummy_task[1285ff04-bccc-46d9-9801-8bc9746abd1c] 
    succeeded in 0.0261888504028s: None

Any help on this would be greatly appreciated. Thanks a lot!

Clay Wardell Clay Wardell · Accepted Answer · 2012-09-05T18:21:27

I sort of figured it out. Because I was debugging, I was naturally setting tasks to run in 5 or 10 minutes. I don't know exactly why, but my set up has a hard time handling this. It's like it needs a certain amount of time to process and queue the task. Anyway, tasks scheduled for more than an hour or so in advance have been running correctly.

Scheduled Celery tasks on RabbitMQ remain unacknowledged past their specified run time

1 Answers