How to structure celery tasks

30
votes

I have 2 types of task: async tasks and schedule tasks. So, here is my dir structure:

proj
  |
  -- tasks
      |
      -- __init__.py
      |
      -- celeryapp.py     => celery instance defined in this file.
      |
      -- celeryconfig.py
      |
      -- async
      |    |
      |    -- __init__.py
      |    |
      |    -- task1.py    => from proj.tasks.celeryapp import celery
      |    |
      |    -- task2.py    => from proj.tasks.celeryapp import celery
      |
      -- schedule
           |
           -- __init__.py
           |
           -- task1.py    => from proj.tasks.celeryapp import celery
           |
           -- task2.py    => from proj.tasks.celeryapp import celery

But when I run celery worker like below, it does not work. It can not accept the task from celery beat scheduler.

 $ celery worker --app=tasks -Q my_queue,default_queue

So, is there any best practice on multiple task files organization?

2
pythoncelery
Have you tried adding -B to the command? It should execute celerybeat - geekazoid
BTW, is it a django project? - geekazoid
Asking about "how to structure celery tasks" is irrelevant/premature unless you know that your file layout is the cause of things not working. Also, please provide more detail on what "it does not work" and "cannot accept the task from celerybeat" mean. In other words, what do you expect to happen, and what happens instead? Specifically. Share error output. - Paul Bissex

2 Answers

10
votes

Based on celery documentation you can import a structure of celery tasks like this:

For example if you have an (imagined) directory tree like this:

|
|-- foo
|    |-- __init__.py
|    |-- tasks.py
|
|-- bar
     |-- __init__.py
     |-- tasks.py

Then calling app.autodiscover_tasks(['foo', bar']) will result in the modules foo.tasks and bar.tasks being imported.

5
votes

Celery tasks can be async, sync or scheduled depends on its invocation

task.delay(arg1,arg2)       #will be async
task.delay(arg1,arg2).get() #will be sync
task.delay(arg1,arg2).get() #will be sync
task.apply_async(args = [arg1,arg2], {'countdown' : some_seconds}) #async with delay

There's a lot of invocations depending on your needs
However, you must start celery with -B flag to enable celery scheduler

$ celery worker --app=tasks -B -Q my_queue,default_queue

So the way you take to organize your tasks is something personal and it deppends on your project complexity, but I think that organize them by its type of synchronism wouldn't be the best option.

I've googled this topic and I haven't found any guide or advise, but I've read some cases that organize their task by their functionality.
I've followed this advise, because this isn't a pattern, in my projects. Here one example of how I did

your_app
    |
    -- reports
        |
        -- __init__.py
        -- foo_report.py
        -- bar_report.py
        -- tasks
            |
            -- __init__.py
            -- report_task.py
    -- maintenance
        |
        -- __init__.py
        -- tasks
            |
            -- __init__.py
            -- delete_old_stuff_task.py
    -- twitter
        |
        -- __init__.py
        -- tasks
            |
            -- __init__.py
            -- batch_timeline.py