Check status of Celery worker

3
votes

I have a project that uses Celery. I am periodically running into a scenario where my requests are making it to Celery but the tasks aren't being handed off to the workers, but rather the server is just returning a 500 error.

When I restart Celery it starts working again. I am only guessing that the worker is hanging which makes it so there aren't anymore workers available. If I startup another batch of workers the requests start working again (which supports my theory).

Questions:

  1. I understand celery by default logs to stderr. I am not seeing any errors, so I am hoping there is another celery log somewhere. Where would that be?
  2. Is there any way to lookup the status of workers? Are they available? Are they hung?
  3. Could it be anything else?
5
celery

5 Answers

12
votes

you can see active workers using flower API or by directly querying celery using below manner:

from celery import Celery
app = Celery('vwadaptor',
             broker='redis://workerdb:6379/0',
             backend='redis://workerdb:6379/0')

app.control.inspect().active()

Shows you the active workers and their currently executing jobs.

1
votes

You can install flower:

pip install flower

It has an API that can be used to ask for the status of the workers and tasks running on them.

0
votes

To get a quick status of your workers you can run this command:

celery -A project inspect stats

it will output a JSON array with lot statistics about your workers.

See docs for more details.

0
votes

I understand celery by default logs to stderr. I am not seeing any errors, so I am hoping there is another celery log somewhere. Where would that be?

Celery logger will write to the file you specified with --logfile=/path/to/file.log. If you did not specify it, it will go, as you say, to stderr.

Is there any way to lookup the status of workers? Are they available? Are they hung?

Yes, it is all covered by the Monitoring and Management Guide of the Celery documentation.

celery -A yourproject.app inspect status will give the status of your workers. celery -A yourproject.app inspect active will give you list of tasks currently running, etc.

Could it be anything else?

Celery does not return any 500 errors. What are you referring to? Celery flower perhaps? - Celery is not a web server... So it is perfectly possible that your Celery cluster works fine, but you are confused with Celery Flower, or some other web server, being down.

0
votes

run this command in your terminal

celery -A yourproject.app  status

for more command

celery -A yourproject.app --help