How to maintain ordering of message in Kafka?

Question

We have requirement of message layer and we want order of messages to be preserved till it gets consumed the consumer. We are exploring Kafka for the same. I understand Kafka doesn't guarantee ordering across partitions but within partition it maintains order.

I was thinking if somehow we can make sure that a particular producer (with certain key value attributes) publishes its messages in a particular partition p1 for topic t1 and if we can make sure that consumer fetches messages from same partition p1 .. it would serve our purpose.

In kafka-console-producer.sh help , I see below:

--producer-property <producer_prop>      A mechanism to pass user-defined
                                           properties in the form key=value to
                                           the producer.
--producer.config <config file>          Producer config properties file. Note
                                           that [producer-property] takes
                                           precedence over this config.
--property <prop>                        A mechanism to pass user-defined
                                           properties in the form key=value to
                                           the message reader. This allows
                                           custom configuration for a user-
                                           defined message reader.

Can we control partition where producer will be writing to , using this --property option?

Also in high level consumer, can we control which partition it will consume from? How partition is assigned to a consumer?

Use common key while producing related events to maintain ordering within a partition: stackoverflow.com/questions/29820384/… — CᴴᴀZ

Tombart Tombart · Accepted Answer · 2017-02-04T11:29:30

Kafka only provides a total order over messages within a partition, not between different partitions in a topic.

If you have a topic with single partition, the ordering is guaranteed. If your consumer if performing well, you don't have to worry.

Assuming you have at least Kafka 0.9, each consumer is assigned a partition when registers at the broker. If you have single topic with 2 partitions and two consumers, each one will be assigned one partition. This information is stored in ConsumerPartitionOwnershipInfo see the design docs for more details. Each partition in the topic is assigned to exactly one member in the consumer group.

Consumer behavior and changes in 0.9 are nicely explained here.

How to maintain ordering of message in Kafka?

1 Answers