Single kafka consumer - reading from mutliple topics - what is the order in which messages will be consumed

1
votes

I am writing a simple Kafka consumer in java, which is configured to read from multiple topics. currently, let's assume two topics ( topic1 and topic 2) and single partition for both topics.

What is the order in which Kafka consumer reads from the topic1 and topic2. If both the topic have let's say 100 messages already published.

Let'say topic1 has the following messages ( A1, A2, A3,......A100) in time ordered fashion.

Let'say topic2 has the following messages ( B1, B2, B3,......B100) in time ordered fashion.

What will be the order of the message received from the topics,

  1. The consumer reads all message from topic1 first followed by topic2?
  2. The consumer reads in time order fashion mixing the messages from both the topics?

I read the Kafka documentation could not find the reference on this. Any help is much appreciated.

1
javaapache-kafkakafka-consumer-api
can you help on how you are doing in scenarios like topic 1 data is parent table data in RDBMS & topic 2 data is child table data, how to we ensure that we dont read child data before parent and get error while Upsert saying Parent not found.. How did you handle multiple topics with parent and child entity but process them correctly without any issues... Did you stage the data and processed the data reading from staging table or did you read the data of parent & child on the fly from Kafka and processed them ? – VR1256

1 Answers

2
votes

How much data consumer will pull on each poll depends on consumer config ( poll records, poll interval ,fetch max bytes ,max partition size etc ) , so there is no guarantee of order of message across the topic partitions , as the ordering is maintained only within a topic partition