Kafka 10.2 new consumer vs old consumer

0
votes

I've spent some hours to figure out what was going on but didn't manage to find the solution.

Here is my set up on a single machine:

  • 1 zookeeper running
  • 3 broker running (on port 9092/9093/9094)
  • 1 topic with 3 partitions and 3 replications (each partition are properly assigned between brokers)

I'm using kafka console producer to insert messages. If i check the replication offset (cat replication-offset-checkpoint), I see that my messages are properly ingested by Kafka.

Now I use the kafka console consumer (new):

sudo bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic testTopicPartitionned2

I dont see anything consumed. I tried to delete my logs folder (/tmp/kafka-logs-[1,2,3]), create new topics, still nothing.

However when I use the old kafka consumer:

sudo bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic testTopicPartitionned2

I can see my messages.

Am I missing something big here to make this new consumer work ?

Thanks in advance.

2
apache-kafka

2 Answers

1
votes

Check to see what setting the consumer is using for auto.offset.reset property

This will affect what a consumer group without a previously committed offset will do in terms of setting where to start reading messages from a partition.

Check the Kafka docs for more on this.

0
votes

Try providing all your brokers to --bootstrap-server argument to see if you notice any differnce:

sudo bin/kafka-console-consumer.sh --bootstrap-server localhost:9092,localhost:9093,localhost:9094 --from-beginning --topic testTopicPartitionned2

Also, your topic name is rather long. I assume you've already made sure you provide the correct topic name.