Spring Kafka Multiple Consumers for a Partition

0
votes

We are using spring kafka to consume messages. We have created receiver for each partition to receive messages. Now we have requirement where we need more than one receivers to consumer message from single partition.

for e.g. Lets suppose we have a Partition 0. Currently we have only one receiver (Receiver 1) that receives messages from this partition. Now I want to add another Receiver (Receiver 2) for the same partition (Partition 0).

So if Producer sends 100 messages to this partition,50 messages should be received by Receiver 1 and remaining 50 messages should be received in Receiver 2. I dont want messages to be duplicated. Both the receiver classes belong to the same group.(I have set props.put("group.id", "unique_group_id") )

Please help me to understand if this scenario is acheivable.

2
apache-kafkakafka-consumer-apispring-kafka

2 Answers

3
votes

NO, you CANNOT do that. Each partition can only be consumed by only one consumer in a single consumer group.

If you want 2 consumers to consume the same partition, these 2 consumers have to belong to 2 different consumer group. However, in this case, both consumers will receive all messages of this partition.

It seems what you need is a worker queue. 2 workers consume messages from a single queue, and each message will be consumed by only one worker. There are many open source worker queues, such as ActiveMQ, RabbitMQ, and so on.

1
votes

If you can't increase number of partitions for the topic, there is some "nasty" trick you can use. Use different consumer-group-id in both receivers and after pool() discard messages with even offset on one receiver and messages with odd offset on the second one, like this:

  R1       R2
| 100      100 [X]
| 101 [X]  101
| 102      102 [X]
| 103 [X]  103
v 104      104 [X]

This will work, but has some consequences:

  • it is not scalable (you cannot easily add another receiver to the partition)
  • if one receiver is stuck/dead, the second one cannot handle it's job
  • offsets management on different consumer groups is quite hard (especially if producers are not under your control)