GCP PubSub retry backoff timing

Question

I have a dead-letter policy configured for my Google Pub/Sub subscription as:

...
  dead_letter_policy {
    dead_letter_topic = foobar
    max_delivery_attempts = x
  }

  {
    "minimumBackoff": y,
    "maximumBackoff": z
  }
...

Plugging in various values, i am not seeing the retries happen at times i would expect. E.g.

max_delivery_attempts: 5 minimumBackoff: 10 Seconds maximumBackoff: 300 Seconds

Seconds between retries: 15 17 20 29

max_delivery_attempts: 30 minimumBackoff: 5 Seconds maximumBackoff: 600 Seconds

Seconds between retries: 12 9 9 14 15 18 24 24 45 44 58 81 82 120 ..., and so on.

From this testing, it seems u need a high max attempts value to get actual exponential back-off? For my first data set, i would have expected the time between my last 2 attempts would have been closer to 300. From my second data set, it seems this would only be the case if the max attempts is set to the max value of 100. Is this assumption correct?

(also, this is a pull subscription)

Thanks

Mahesh Gattani Mahesh Gattani · Accepted Answer · 2021-03-25T20:21:48

Related answer: How does the exponential backoff configured in Google Pub/Sub's RetryPolicy work?

The exponential backoff based on minimum_backoff and maximum_backoff roughly follows the equation mentioned in the question above (with randomization factor). The relevant factor to your question are

Maximum backoff is not part of the calculation when deriving the backoff interval. Maximum backoff setting is used to ensure we do not back off more than configured, even if the backoff interval computation results in such an answer. The rate of growth in interval duration still increases with retry, as visible from your test.
The multiplication factor, responsible for the growth in backoff interval, is a system internal detail and clients should not be dependent on it.

If you want the maximum backoff to happen before the dead letter event occurs, I suggest starting with a higher minimum backoff configuration.

GCP PubSub retry backoff timing

1 Answers