Does Google Cloud (GCP) Pub/Sub supports feature similar to ConsumerGroups as in Kafka

Question

Trying to decide between Google Cloud (GCP) Pub/Sub vs Manager Kafka Service.

In latest update, Pub/Sub added support to replay messages which were processed before, is a welcome change.

One feature I am not able to find on their documentation is whether we can have something similar to Kafka's Consumer Groups, i.e have group of Subscribers and each processing data from the same topic, and be able to re-process the data from beginning for some Subscriber(consumer group) while others are not affected by it. eg:

Lets say you have a Topic as StockTicks

And you have two consumer groups

CG1: with two consumers
CG2: With another two consumers

In Kafka I can read messages independently between these groups, but can I do the same thing with Pub/Sub.

And Kafka allows you to replay the messages from the beginning, can I do the same with Pub/Sub, I am ok if I cant replay the messages that were published before the CG was created, but can I replay the message that were submitted after a CG/Subscribers were created?

Answer 1

Cloud Pub/Sub's equivalent of a Kafka's consumer groups is a subscription. Subscribers are the equivalent of a consumer. This answer spells out the relationship between subscriptions and subscribers in a little more detail.

Your example in Cloud Pub/Sub terms would have a single topic, StockTicks, with two subscriptions (call them CG1 and CG2). You would bring up four subscribers, two that fetch messages for the subscription CG1 and two that fetch messages for the CG2 subscription. Acking and replay would be independent on CG1 and CG2, so if you were to seek back on CG1, it would not affect the delivery of messages to subscribers for CG2 at all.

Keep in mind with Cloud Pub/Sub that only messages published after a subscription is successfully created will be delivered to subscribers on that subscription. Therefore, if you create a new subscription, you won't get the all of the messages published since the beginning of time; you will only get messages published from that point on.

If you seek back on a subscription, you can only get up to 7 days of messages to replay (assuming the subscription was created at least 7 days ago) since that is the max retention time for messages in Cloud Pub/Sub.