Google Cloud PubSub throws 504 Deadline Exceeded error while listening for messages in asynchronous pull

Question

I have a service that subscribes to a PubSub topic and uses the asynchronous pull. After an idle of 10 minutes and not receiving any messages, PubSub throws an 504 Deadline exceeded error.

Error always occurs after about 10 minutes. Every similar issue I found was related to synchronous pull, not the asynchronous pull I use.

Error message:

INFO:google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager:Observed non-terminating stream error 504 Deadline Exceeded                                                                                                           
INFO:google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager:Observed recoverable stream error 504 Deadline Exceeded                                                                                                                 
INFO:google.api_core.bidi:Re-established stream                                                                         
INFO:google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager:Observed non-terminating stream error 504 Deadline Exceeded                                                                                                             
INFO:google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager:Observed recoverable stream error 504 Deadline Exceeded

Subscriber Class:

from google.cloud import pubsub_v1
class Subscriber():

    def __init__(self):
        self.subscriber = pubsub_v1.SubscriberClient()
        self.project_id = "my-project-id"
        self.subscription_id = "SUBSCRIPTION_MAIL_RECIEVE"
        self.subscription_path = self.subscriber.subscription_path(self.project_id,
                                                                   self.subscription_id)

    def subscribe(self, callback):
        """
        Parameters:
            callback (callable): function to be called for incoming messages on given topic
        """
        streaming_pull_future = self.subscriber.subscribe(self.subscription_path,
                                                          callback=callback)
        return streaming_pull_future

Listening for messages:

subscriber = Subscriber()
pull_future = subscriber.subscribe(my_callback_function(message))

with subscriber.subscriber:
        try:
            print("Waiting for messages")
            pull_future.result()
        except TimeoutError:
            pull_future.cancel()

With 10 minutes idle you mean that you dont send any messages to your topic in this period? — rmesteves
Yes, exactly. No messages get published to the topic and my service tries to listen to an "empty" topic. — kayn99
Have you tried using streaming_pull? googleapis.dev/python/pubsub/latest/subscriber/api/client.html — rmesteves

Ankush Handa Ankush Handa · Accepted Answer · 2020-07-15T11:43:10

This is the normal behaviour and these messages can be ignored as your stream is getting re-established. If in case of the stream is not getting re-established and heartbeat is not sent, then probably you need to upgrade your google core API and google cloud Pub/Sub to latest versions.

pip install google-cloud-pubsub --upgrade

pip install google-api-core --upgrade

You can also filter these error messages as suggested in link https://github.com/googleapis/google-cloud-python/issues/5800

Google Cloud PubSub throws 504 Deadline Exceeded error while listening for messages in asynchronous pull

1 Answers