Apache airflow celery workers on localhost

0
votes

How many apache airflow celery workers can I start on localhost? I guess the default setting should be one celery worker that work concurrently by multi-threading. The concurrency can be set in airflow.cfg file (default is 16).

I checked the flower server and there is only one celery worker.

Is it possible to add another celery worker to my localhost?

1
celeryairflow
I believe there's no limit imposed by Airflow in this regard. Celery might cap the number of workers on a machine, but even that seems a remote possibility. You need to play around with 1. Celery (more likely) 2. Airflow configs, and I think this can be pulled off - y2k-shubham
@y2k-shubham. I also thought there's no limit, but how to run a second celery worker on localhsot, that would be my question. I don't think the configure file has something related, may be I will check source code. - Hugo Liu
Oh that? Did you check puckel/docker-airflow? - y2k-shubham
I deploy my airflow and celery on my Ubuntu localhost, not using docker - Hugo Liu

1 Answers

0
votes

I can actually start as many celery workers with cmd

airflow worker -cn [hostname]