I have a design question. I am using dockerized celery workers on several hosts. I only have one instance of the celery container running on each host but using the default workers settings for celery which defaults to the number of cores on that host. I did not set any limits for the docker containers. I used rancher to deploy to the hosts using cattle environment but I guess my question is equally applicable to any docker clustering like swarm. I did not use the scaling features by using more than one container because of the way celery works-one container is already able to leverage the cores by having multiple workers. The question is: Are there any benefits for me to have more 1 worker container on the host? If so, would I need to limit each celery worker to just one in each container and let the cluster to scale multiple containers? The only benefit I can imagine is from a high availability perspective that if the celery worker dies on on host then it is gone, but if I have more containers other can take over the work, but I think celery can do the same thing by respawning workers too. Am I missing something?
1 Answers
0
votes
The only way to know for sure is to benchmark it with your particular workload but your intuition is generally correct. If the application is capable of consistently using all the cores then running more of them will generally make things slightly slower because of context switching. Side benefits like still being available I'd I've of many workers fail may or may not be worth the overhead to you.
