Order of messages from multiple topics Kafka

2
votes

I'm developing a software that uses Apache Kafka. I've got one consumer that subscribed to multiple topics, I'd like to know if there is an order for receiving messages from those topics. I tried some combination on my computer but I need to be sure about this. Example

  • Consumer sub to topic1 and topic2
  • Producer1 write something on topic1
  • Producer2 write something on topic2
  • Producer1 write something on topic1
  • When the consumer polls, it receives a list of records containing first the messages from the first topic that he subscribed and then the messages from the other topic.

I'd like to know if it is always like this, i.e. the messages are in order like the topics that I subscribed. Thanks

[EDIT] I'd like to specify that I have the two topics with one partition each, and only one producer and one consumer. I need to read first all the messages from the first topic and then the messages from the other topic

3
javaapache-kafkakafka-consumer-api
Kafka represents an infinite message queue, how will you define that you are done with consuming all message from topic1? – Rahul Shukla

3 Answers

2
votes

Kafka gives you only the guarantee of messages ordering inside a partition. It means that even with only one topic but more than one partitions you have no guarantee that messages are received in the same order they are sent.

Regarding your use case with two topics there is no relation between subscription order to the topics and messages ordering even because if the cluster has more than one node, the topic partition leader will be on different brokers and the client receives messages over different connections. Btw even with only one broker with all topics/partitions on that you can't have the guarantee you are describing.

1
votes

No. Message ordering is only preserved within partitions (not even within topics).

If you need stronger ordering guarantees, you have to re-arrange messages in your application, for example using a timestamp (and a sufficiently large window buffer to catch all the ones that arrive out-of-order). Support for this has improved a bit with the recent addition of timestamps for all messages by Kafka itself, but the principle remains the same.

0
votes

Why not first subscribe to the first topic and do a poll, and then subscribe to the other topic and do another poll? Without this, I don't think there is any guarantee in which order you receive messages from the two topics.