GCloud Pub/Sub Push Subscription: Limit max outstanding messages

6
votes

Is there a way in a push subscription configuration to limit the maximum number of outstanding messages. In the high level subscriber docs (https://cloud.google.com/pubsub/docs/push) it says "With slow-start, Google Cloud Pub/Sub starts by sending a single message at a time, and doubles up with each successful delivery, until it reaches the maximum number of concurrent messages outstanding." I want to be able to limit the maximum number of messages being processed, can this be done through the pub/sub config?

I've also thought of a number of other ways to effectively achieve this, but none seem great:

  • Have some semaphore type system implemented in my push endpoint that returns a 429 once my max concurrency level is hit?

  • Similar, but have it deregister the push endpoint (turning it into a pull subscription) until the current messages have been processed

My push endpoints are all on gae, so there could also be something in the gae configs to limit the simultaneous push subscription requests?

1
google-cloud-platformgoogle-cloud-pubsub
You can see in docs here: cloud.google.com/pubsub/docs/subscriber that you can limit the amount of time for messages to be acknowledged: "A subscriber has a configurable, limited amount of time, or ackDeadline, to acknowledge the message. Once the deadline has passed, an outstanding message becomes unacknowledged." Can you use something like that? - VictorGGl

1 Answers

2
votes

Push subscriptions do not offer any way to limit the number of outstanding messages. If one wants that level of control, the it is necessary to use pull subscriptions and flow control.

Returning 429 errors as a means to limit outstanding messages may have undesirable side effects. On errors, Cloud Pub/Sub will reduce the rate of sending messages to a push subscriber. If a sufficient number of 429 errors are returned, it is entirely possible that the subscriber will receive a smaller number of messages than it can handle for a time while Cloud Pub/Sub ramps the delivery rate back up.

Switching from push to pull is a possibility, though still may not be a good solution. It would really depend on the frequency with which the push subscriber exceeds the desired number of outstanding messages. The change between push and pull and back may not take place instantaneously, meaning the subscriber could still exceed the desired limit for some period of time and may also experience a delay in receiving new messages when switching back to a push subscriber.