pubsub Dynamic rate limiting

Question

Can anyone give details on the Dynamic rate limiting implemented by the Pub/Sub system? I couldn't find any details on the gcloud docs or the faq pages.

Here is my pubsub usage: I'm planning to use pubsub in our production. Right now, I have 1 topic, 1 subscription and 1 subscriber (Webhook HTTPS callback). Sometimes my subscriber can throw an exception (very rarely), in that situation my subscriber shall return a 400 response back to the pubsub, so that the pubsub can retain the message and retry.

If the pubsub gets a 400 response from the subscriber, will it severely impact the flow rate of other messages? Given the scarce documentation on how the flow control is implemented, i'm mainly concerned about the impact of one bad message on latencies of all other good messages.

I can split my one topic into multiple topics and multiple subscriptions, if it helps reduce the impact of a bad message.

Answer 1

If you are only occasionally returning a 400, you should not see a severe impact on the rate of messages delivered to your subscriber. When a 400 response occurs, as mentioned in the Subscriber Guide, the number of allowed outstanding messages would be cut in half. If you then return success for another outstanding message, the window will be immediately doubled again, effectively not reducing the number of outstanding messages allowed.

Message delivery for subsequent messages is delayed by an amount that is exponentially increasing on subsequent failures, starting with a delay that is O(10s of ms). Whenever a success response is returned, subsequent messages are no longer delayed. Therefore, a single 400 response from a subscriber that is otherwise returning successes shouldn't really have any noticeable impact.

Answer 2

Messages in Pub/Sub are retained until the consumer acknowledges the message. As long as the consumer does not acknowledge that it processed the message, the message will be retained and re-delivered.