Kafka mirror maker putting all messages without a partitioning key together randomly in the target cluster

3
votes

We are using Kafka mirror maker to mirror topics across a 0.8.2.1 cluster to a 0.10.2.0 cluster. However for certain topics we are relying on a round robin partition assignment ( no explicit partitioning keys ).

I know mirror maker preserves the key and maintains the partitions from source to target but I am seeing something weird happening:

source cluster offsets:

event_edi:0:28731
event_edi:1:28775
event_edi:2:28643

destination cluster offsets:

event_edi:2:169
event_edi:1:88
event_edi:0:85596

Most of these messages did not have any keys . It seems weird that they were all grouped together and dumped in the same partition instead of round robin.

For mirror maker I have a SyncProducer running and rest of the configs are all default.

If someone can help me understand how this can be avoided / or if I have it all wrong will be grateful! TIA!

1
apache-kafkakafka-consumer-apikafka-producer-api
Did you get any help, tips, solution for this since? Or have you found the root problem? – David
I think if you do not explicitly partition your logs with a key , it does this. Our messages did not have any partitioning key . – GothamGirl

1 Answers

0
votes

If you do not explicitly partition your logs with a key , it does this. Our messages did not have any partitioning key and hence landed in the same partition