I read following in kafka docs:
- The way consumption is implemented in Kafka is by dividing up the partitions in the log over the consumer instances so that each instance is the exclusive consumer of a "fair share" of partitions at any point in time.
- Kafka only provides a total order over records within a partition, not between different partitions in a topic.
- Per-partition ordering combined with the ability to partition data by key is sufficient for most applications.
- However, if you require a total order over records this can be achieved with a topic that has only one partition, though this will mean only one consumer process per consumer group.
I read following on this page:
- Consumers read from any single partition, allowing you to scale throughput of message consumption in a similar fashion to message production.
- Consumers can also be organized into consumer groups for a given topic — each consumer within the group reads from a unique partition and the group as a whole consumes all messages from the entire topic.
- If you have more consumers than partitions then some consumers will be idle because they have no partitions to read from.
- If you have more partitions than consumers then consumers will receive messages from multiple partitions.
- If you have equal numbers of consumers and partitions, each consumer reads messages in order from exactly one partition.
Doubts
Does this means that single partition cannot be consumed by multiple consumers? Cant we have single partition and a consumer group with more than one consumer and make them all consume from single partition?
If single partition can be consumed by only single consumer, I was thinking why is this design decision?
What if I need total order over records and still need it to be consumed parallel? Is it undoable in Kafka? Or such scenario does not make sense?