Cloud Pub/Sub missing ack/nack in callback not causing redelivery

Question

I'm trying to make sure that Cloud Pub/Sub will redeliver my message if there is no ack/nack sent. It doesn't seem to do this even though I wait for more than 10 minutes, which should be the maximum time for the Acknowledgement Deadline.

I'm using the example here as a starting point: https://cloud.google.com/pubsub/docs/quickstart-py-mac

Essentially, I commented the line in the callback function that ack's the message. I used two terminals, one for publishing the message and one for receiving it as a subscriber. Since no ack's are being sent, I expected Cloud Pub/Sub to attempt to redeliver the message to the subscriber sometime within the ack deadline, but it doesn't.

The docs here

https://godoc.org/cloud.google.com/go/pubsub#hdr-Deadlines

say the the "ACK deadlines are extended periodically by the client... up to 10 minutes", so I waited 10 minutes in case the ack deadline was extended to that maximum but I still didn't receive the redelivered message.

Here is the edited callback method that I used. This was the only change I made to the example code.

def callback(message):
        print('Received message {} of message ID {}'.format(
            message, message.message_id))
        # Acknowledge the message. Unack'ed messages will be redelivered.
        # message.ack()
        print('Acknowledged message of message ID {}\n'.format(
            message.message_id))

If I kill the subscriber (sub.py) and restart it, the message is redelivered. Am I doing something wrong? Also, when I send a Nack instead of not sending anything at all, the message is quickly redelivered.

EDIT:

It seems like similar questions have been asked on

https://github.com/googleapis/google-cloud-python/issues/5005

https://github.com/googleapis/google-cloud-python/issues/5044

Things I wanted to confirm:

The Ack Deadline set in the subscription is not always the value used. It's extended as Pub/Sub deems necessary.
The 10 minute max Ack Deadline is not actually the maximum time that can elapse before a message is redelivered
This maximum time is determined by the flow_control.max_lease_duration variable (default is 2 hours)

Add this to the bottom of your code return abort(500). Is the message redelivered? — John Hanley
@JohnHanley Adding return abort(500) to the end of my callback method causes this error: 'No handlers could be found for logger "google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager"', then immediate, continuous redelivery of the message roughly every half second. The issue I'm concerned about is if for some reason the callback halts or errors and the ack/nack is never sent (or in this case, the abort) — Gavin Lee
I'm not sure if this helps, but I ran the subscriber for a couple hours and after reaching this error, "RetryError: Deadline of 600.0s exceeded while calling <functools.partial object at 0x1074758e8>, last exception: 503 failed to connect to all addresses", the messages started being redelivered every 10-12 seconds. Maybe that caused the ack deadline to be reset to the default? — Gavin Lee

Kamal Aboul-Hosn Kamal Aboul-Hosn · Accepted Answer · 2019-07-22T10:46:39

The ackDeadline is the maximum amount of time a message can be outstanding to a subscriber without ack, nack, or modAckDeadline. Once this time passes, the message becomes a candidate to be redelivered, though redelivery may not be immediate. The maximum value for this field is ten minutes.

The Cloud Pub/Sub client libraries, upon receiving a message, automatically sends modAckDeadline requests for the message until it is acked, nacked, or the flow_control.max_lease_duration period passes. Essentially, it extends the ack deadline. The goal is for the client library to keep track of ack deadlines and extend messages as necessary itself so the user doesn't have to do it.

This is why you see the discrepancy between the client library behavior and the ackDeadline configured on the subscription. When using the client libraries, you should set the max_lease_duration to the maximum amount of time you want a message to be outstanding to a subscriber.

Cloud Pub/Sub missing ack/nack in callback not causing redelivery

1 Answers