How to make consumer group A pick up offset of consumer group B but without affecting A's committed offset?

1
votes

I created a topic "Item" and it has 8 partitions.

Also, I have two consumer groups A and B both are listening to this topic. Consumer group A has offset on each of the partition.

However, I would like to set consumer group B's offset same as consumer's A's offset so that consumer B can start to consume messages from where A left without affecting A's committed offsets. Say, I have a database that store consumer A's offset and I would like B to consume from there.

Let me add a use case: Group A is inserting messages into a table in database along with offsets. However, at some point, I copied the table to another database. But I copied a snapshot (say snapshot from 10 mins ago) of this table so some trades inserted in last 10 mins are not there. In this case, I want to use Group B to populate missed trades in the copied table. So I would like Group B to set its offsets to the offsets stored on the last inserted item in that table.

Is there a way to achieve this ? Please correct me if I misunderstood something.

1
apache-kafkakafka-consumer-api
Will consumer A continue to consume messages after consumer B started consuming from consumer A's offsets? – Michael Heil
Why don't you just create one cosnumer group with more consumers? – Giorgos Myrianthous
@mike A will continue to consume messages. But B shouldn't commit offset on Group A. I added a use case. – Guifan Li
@GiorgosMyrianthous I added a use case. More consumers won't solve my issue. – Guifan Li

1 Answers

1
votes

You could manually change the offsets of the Consumer Group B using the ConsumerGroup Command tool. You can find some information in the Kafka documentation Managing Consumer Groups.

If you plan to reset a particular Consumer Group ("consumerB") you can use

> bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --reset-offsets --group consumerB --topic Item --to-latest

There are also options available for each individual partition. That way you could set the offset to a particulat offset of consumer A.