Application impacts of celery workers running with the `--without-heartbeat` flag

2
votes

Discussion here talks high level about some of the impacts of running celery workers with the --without-hearbeat --without-gossip --without-mingle flags.

I wanted to know if the --without-heartbeat flag would impact the worker's ability to detect broker disconnect and attempts to reconnect. The celery documentation only opaquely refers to these heartbeats acting at the application layer rather than TCP/IP layer. Ok--what I really want to know is does eliminating these messages affect my worker's ability to function--specifically to detect broker disconnect and then to try to reconnect appropriately?

I ran a few quick tests myself and found that with the --without-heartbeat flag passed, workers still detect broker disconnect very quickly (initiated by me shutting down the RabbitMQ instance), and they attempt to reconnect to the broker and do so successfully when I restart the RabbitMQ instance. So my basic testing suggests the heartbeats are not necessary for basic health checks and functionality. What's the point of them anyways? It's unclear to me, but they don't appear to have impact on worker functionality at the most basic level.

What are the actual, application-specific implications of turning off heartbeats?

1
pythoncelery

1 Answers

2
votes

So this is the explanation of the heartbeat mechanism. Now since AMQP uses TCP the celery workers will try to reconnect if they can't establish a connection or whenever the TCP protocol dictates. So it looks like the heartbeat mechanism is not needed. But it as a few advantages :

  1. It doesn't rely on the broker protocol, so if the broker have some internal issues or uses UDP the worker will still know if events are not received, and will be able to act accordingly

  2. The heartbeat mechanism checks that events are sent and received which is a much greater indicator that the app is running as expected. If for example the broker doesn't have enough space and is starting to drop events, the worker will have an indication for that with the heartbeat mechanism. And if the worker is using multiple brokers, it can also decide to connect to another broker which should be less busy.

NOTE: regarding "[heartbeat] does not rely on the broker protocol... or uses UDP":

  • Given that celery supports multiple brokers and may use UDP, celery wants to guarantee the connection to the broker even if your broker protocol uses UDP --> so the only way for celery to guarantee that connection when the broker protocol uses UDP is to implement your own application level heartbeats.