Check if there are running tasks on an airflow worker (celery)

1
votes

I want to check (programmatically) if any tasks are running on a specific celery worker. I don't care where the solution should be executed, it can be on the airflow-scheduler/db machine or on the airflow worker machine itself.

I've checked this: How do I check if there are DAGs running in Airflow (before restarting Airflow)? However this will just check for running tasks across all workers. I want to check if a specific worker has no running tasks so I can stop the worker (downscale workers).
I have flower installed as well, and I can monitor Succeeded/Failed tasks but I'm not sure those help me.
Queues are not used but they can be if needed.
Can I monitor the processes to see if their parent is airflow worker/celery or something ?

Any ideas ?

2
celeryairflowairflow-worker

2 Answers

2
votes
  1. You can see more than that in Flower (just click on a specific worker): enter image description here The data is also available via rest API - as you can see in the docs:
GET /api/workers HTTP/1.1

  1. Not sure what's your Celery's broker is - if it's RabbitMQ, you can use Prometheus metrics to get the queue size (I wrote a post about this topic).

  2. Celery support autoscale to resize the pool size, it that's help.

  3. You can inspect the celery's workers by code and get active tasks per worker.

  4. I never tried, but from the ref you added, seems like the task_instance table has external_executor_id column - maybe that's the worker id?

1
votes

The simplest thing (and something I do every day) is to execute something like celery -A your.project inspect active -d <name of your celery node> (this works with Airflow too, when Celery executors are used).

With regards to shutting down the worker. - There is no need to wait for worker to have no running tasks to send a warm shutdown signal to it. When you do so, worker will unsubscribe from its queues and wait for all worker processes to finish before it shuts down.

If you plan to do a cold shutdown though, then you do need something like what you plan to do, but there is always a chance that the worker will start a task right before you executed the cold shutdown...

Queues are not used but they can be if needed.

Well, not entirely true as the default queue is used by default...