GKE: Pub/Sub and Pod deplyment with 3 replicas

Question

In GKE, if i have a Pub/Sub topic that is set up to use the pull method and an API that acts as the subscriber to this topic, if this API has a replication of 3 (spec.replicas: 3), what is the out of the box behaviour for the API (client)?

i.e. when a message gets pushed to the topic, given the API is asynchronously pulling messages from topic (https://cloud.google.com/pubsub/docs/pull#asynchronous-pull) and has replication of 3, do all 3 pods pull for the message at the same time (and ending up with duplicates)? Is there some kind of load balancing behind the scenes? what's the out of the box behaviour?

Answer 1

You have a great video series on youtube: the Google Cloud youtube channel. You can understand deeply the behavior of each type of subscription, the retry polices and so on.

To answer immediately your question: Yes there is a load balancer and a messages are dispatched according with the number of subscriber. But it's not really a round robin. There is under the hood optimization that dispatch chunk of messages to each subscriber (according with their size and number). I mean, if you send 3 test messages in the same time, the 3 will go to the same subscriber.

The messages are only duplicated in subscription (push or pull).

Answer 2

Messages are load-balanced across subscriber clients that connect with the same subscription. A given message will only be outstanding to one subscriber at a time, until the ack deadline expires, at which point it may be redelivered to a different subscriber client. The service also respects flow control settings for each subscriber client individually and will not send message to a client that is flow controlled, instead routing them to other clients. More information about subscriber behavior is available in the subscriber documentation.

So if 3 replicas connect with the same subscription, the messages of the topic will be load balanced across them i.e. they will receive different subsets of messages. You should provision enough replicas so that in aggregate they can process quicker than messages are published to the topic.

If you want 3 replicas to each receive all messages separately, use a different subscription for each replica.