Kafka - how is a SimpleConsumer associated to a consumer group?

0
votes

New to Kafka.

I'm really confused by Kafka's API:

  • Version 0.9 is completely different from 0.8.

  • Then there are the simpleConsumer, the highlevel Consumer and the consumer group

    1. When I instantiate a SimpleConsumer is it associated to a consumer group? Or is the consumer group an abstraction which is used by the high-level consumer?
    2. If I don't care about ordering of messages or duplicates, can I instantiate 2 simpleConsumers that read from the same partition?
    3. Is there a way to use a simpleConsumer to read from the topic without specifying partitions?
1
javaapache-kafka

1 Answers

1
votes

With Kafka 0.9 there is a new consumer API as you noted and the two older consumer APIs still exist but will likely be decommissioned in a future release in favour of the new API.

The consumer group concept relates only to the high-level consumer and is a helper to coordinate consumer instances reading from the same set of topics to avoid duplicated messages and allow parallelism with automatic fail-over in case of a consumer instance crash etc. When using the simple consumer API, you have to take care of this coordination yourself and therefore you also need to specify which partitions to read from and it's also not preventing you from having multiple consumers reading from the same partition.

I don't know of a good use case where you would need multiple consumers reading from the same partition though, if you want to consume it for different purposes, you can just use the high-level API with multiple consumer group IDs and they would work independently from each-other.