I am trying to get my head around possible consumer patterns with Kafka 0.8.1.1. Let's ignore replication because I dont believe it affects these patterns. I ran the command line console consumer for all my tests.
Can you please confirm that my understanding is correct?
1) One topic, many partitions m, many consumers n, m=>n, no groups defined.
In this case every consumer will get every message on the topic.
2) One topic, many partitions m, many consumers n with m LESS THAN n , no groups defined.
Same behavior. Every consumer will get every message on the topic.
3) One topic, many partitions m, many consumers n, m LESS THAN n , one consumer group defined for all the consumers
I see "No broker partitions consumed by consumer thread" on one of the consumer consoles. Is this because there are more consumers than partitions? ( In this case m=3, n=4 )
4) One topic, many partitions m, many consumers n, m=n , one consumer group defined for all the consumers
From using Kafka Monitor, I see that each partition is assigned to one consumer now. However, it seems that there is no parallelism in data consumption. What I see happening is that one consumer gets messages from time t0 to t1 from partition P1. Then another consumer gets messages from t1 to t2 from partition P2 and so on.
Why is there no parallel consumption happening ? It looks to me that the producer's data goes into P1 from t0 to t1 and then from t1 to t2 into P2. I thought that if I dont specify a partitioning key, the producer's data will get partitioned randomly. It's just that the randomness seems to be "delayed". Why is this so ?
