Multiple Python-Celery Scripts conflict and doesn't execute

Question

I have two different python scripts in different directories that has celery schedulers.

Script 1:

import requests
from celery import Celery
from celery.schedules import crontab
import subprocess

celery = Celery()
celery.conf.enable_utc = False


@celery.task()
def proxy():
    response = requests.get(url="XYZ")
    proxies = response.text

    paid_proxies = open("paid_proxies.txt", "w+")
    paid_proxies.write(proxies.strip())
    paid_proxies.close()


celery.conf.beat_schedule = {
    "proxy-api": {
        "task": "scheduler1.proxy",
        "schedule": crontab(minute="*/5")
    }
}

Commands that I use for executing it:

celery beat -A scheduler1.celery
celery worker -A scheduler1.celery

Script 2:

from celery import Celery
from celery.schedules import crontab
import subprocess

celery = Celery()
celery.conf.enable_utc = False


@celery.task()
def daily():
    subprocess.run(["python3", "cross_validation.py"])


celery.conf.beat_schedule = {
    "daily-scraper": {
        "task": "scheduler2.daily",
        "schedule": crontab(day_of_week="*", hour=15, minute=23)
    }
}

Commands that I use for executing it:

celery beat -A scheduler2.celery
celery worker -A scheduler2.celery

The issues is when I execute Script 1, it works perfectly. But when I try to execute Script 2, I get this error as Scheduler2 tries to execute tasks of scheduler1:

[2019-09-14 15:10:00,127: ERROR/MainProcess] Received unregistered task of type 'scheduler1.proxy'. The message has been ignored and discarded.

Did you remember to import the module containing this task? Or maybe you're using relative imports?

Please see http://docs.celeryq.org/en/latest/internals/protocol.html for more information.

The full contents of the message body was: '[[], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]' (77b) Traceback (most recent call last): File "/home/PycharmProjects/data_scraping/venv/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 559, in on_task_received strategy = strategies[type_] KeyError: 'scheduler1.proxy'

I tried referring multiple answers but nothing worked.

2ps 2ps · Accepted Answer · 2019-09-14T14:46:18

The problem that you are seeing is that celery is using the same "broker" in both project 1 and project 2. In order to use two different celery projects simultaneously, all you have to do is give them different brokers. You can specify a broker using the broker_url setting.

We typically use redis as a broker, so it is very simple to put one project on redis db 0 and the other project on redis db 1. That said, there is a lot of thinking that normally goes into which broker to use, and deciding on a broker is outside the scope of this particular question.

Multiple Python-Celery Scripts conflict and doesn't execute

1 Answers