Before consumers for new topic are attached, I create new topic and produce message in apache kafka

Question

Before consumers for new topic are attached, I create new topic and produce a first message in apache kafka. Then consumers for new topic are attached, but the first message could not be consumed. Why..?

In this case, already log-end offset=1, commited offset=1, lag=0.

Doesn't "commited offset=1" mean it's already been consumed? My question is why it has already been consumed. Let me know if there's anything I'm wrong with.

This is my test case.

# create new topic 
$ kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic NEW_TOPIC_NAME

# produce a first message
$ kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic NEW_TOPIC_NAME
  > send a first message

# then execute consumer
$ kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic NEW_TOPIC_NAME
  >   # no consume a first message

But after consumers for new topic are attached, I produce a second message then normally consume.

The offset being 1 on the consumer does not mean that the message has been consumed. The offset indicates where in the topic the consumer will begin reading messages from. — brad mcallister

Mickael Maison Mickael Maison · Accepted Answer · 2018-07-05T09:22:59

By default, the kafka-console-consumer starts from the end of the topic.

If you want to consume messages produced before, you can set --from-beginning for example:

kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 
  --topic NEW_TOPIC_NAME --from-beginning

Before consumers for new topic are attached, I create new topic and produce message in apache kafka

1 Answers