Celery Daily Scheduled Tasks with Crontab

Question

I have problem with daily scheduled tasks with crontab. Here is my celery.py

app.conf.beat_schedule = {
    'run-cache-updater': {
        'task': 'tasks.run_cache_updater',
        'schedule': crontab(
            minute=0,
            hour='1-4'
        ),
    }
}

Below is my tasks.py What I am doing there is, getting all records from DB. Triggering other jobs to update my caches on Redis.

@app.task
def run_cache_updater():
    batch_size = 1000
    cache_records = models.CacheRecords.objects.all()

    def _chunk_list(all_records, size_of_batch):
        for i in range(0, len(all_records), size_of_batch):
            yield [item.id for item in all_records[i: i + batch_size]]

    for items in _chunk_list(cache_records, batch_size):
        update_cache.delay(items)


@app.task
def update_cache(ids_in_chunks):
    for id in ids_in_chunks:
        # Some calls are done here. Then sleep for 200 ms.
        time.sleep(0.2)

My tasks are running good. However, they start to run between 1 and 4 and then they start again every 4 hours like 8-11, 15-18.. What I am doing wrong here and how can I fix it?

Symmetric Symmetric · Accepted Answer · 2018-09-04T02:01:39

This sounds like a Celery bug, it's probably worth raising on their Github repo.

However, as a workaround, you could try the more explicit notation, hour='1,2,3,4', just in case the issue is in the parsing of that specific crontab interval style.

Celery Daily Scheduled Tasks with Crontab

1 Answers