Kafka Topic ordering when scaling up the partitions

2
votes

Consider your producers create messages for the users of a system and the order of them is important in the user level.

My producers, add messages to the topic which have two partitions and I am using hashing against the user_id to put all the messages of each user in the same partition to guarantee the order.

How can I scale up the system and add more partitions to the topic while keeping the order of the messages?

How Kafka treat the messages that are already produced before partitioning? What will happen to the messages that consume but not committed back to the Kafka to update the offset?

1
apache-kafkakafka-consumer-apiproducer-consumer
Regarding old messages you can check it: stackoverflow.com/questions/54804205/… – Bartosz Wardziński

1 Answers

0
votes

1.use a treeset(ordered set) cache messages at consumer client, keep 1 minute(or less); kafka only guarantee one partition's order, and I think producer also cannot guarantee order。

2.if you not commit offset manually, in the next fetch request ,will get same message. anyway, at consumer client, you should ensure message idempotency, even you conmmited offset.