kafka console consumer not receiving messages from console producer

Question

I recently setup a cloudera quickstartVM using docker image and setup Kafka parcel in it. After successful installation, i see that all the services are running in green status (including Kafka and zookeeper). However, when I follow the below commands of kafka CLI i don't see consumer getting messages.

Any help is greatly appreciated CDHv 5.13 CDK 4.0 (kafka 2.1)- through parcel.

kafka-topics --create quickstart.cloudera:9092 --replication-factor 1 --partitions 1 --topic test3 --zookeeper quickstart.cloudera:2181

topic created successfully on console.

Console Consumer (CLI terminal 1):
kafka-console-consumer --bootstrap-server quickstart.cloudera:9092 --topic test3

consumer started on console in Terminal 1

Console Producer:
kafka-console-producer --broker-list  quickstart.cloudera:9092 --topic test3

Producer created in Terminal 2. Now, when i type anything in console of Terminal 2 (producer), the consumer terminal doesn't show anything.

Please suggest what is missing here. I am not sure how to debug this situation.

I don't see any exception in the /var/log/kafka/kafka-broker-quickstart.cloudera.log file

19/05/07 09:18:28 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /172.17.0.2:60968, server: quickstart.cloudera/172.17.0.2:2181
19/05/07 09:18:28 INFO zookeeper.ClientCnxn: Session establishment complete on server quickstart.cloudera/172.17.0.2:2181, sessionid = 0x16a915bca140112, negotiated timeout = 30000
19/05/07 09:18:28 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Connected.
Topic:test4     PartitionCount:1        ReplicationFactor:1     Configs:
        Topic: test4    Partition: 0    Leader: 37      Replicas: 37    Isr: 37
19/05/07 09:18:28 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Closing.
19/05/07 09:18:28 INFO zookeeper.ClientCnxn: EventThread shut down
19/05/07 09:18:28 INFO zookeeper.ZooKeeper: Session: 0x16a915bca140112 closed
19/05/07 09:18:28 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Closed.

This is the output of

kafka-topics --zookeeper quickstart.cloudera:2181 --describe --topic test3

Sometimes you use test3 and sometimes test4. And your creation command looks like an incorrect mix of using --bootstrap.server and --zookeeper (newer versions of kafka use --bootstrap.server, but I don't know for 2.1)... — pgras
thanks. I fixed the typo. I am following the instructions provided in the below link cloudera.com/documentation/kafka/latest/topics/… — Prashant
The create command you have in the question is different from the one from your link (not only the zookeeper address), what happens when you run kafka-topics --zookeeper zk01.example.com:2181 --list (with the address of your ZooKeeper) ? — pgras
Maybe you could try to follow that example kafka.apache.org/documentation.html#quickstart_createtopic and adapt to your cluster address. — pgras

Prashant Prashant · Accepted Answer · 2019-05-07T12:30:51

I removed the CDK4.0 parcel and instead added CDK 3.1 parcel. Now, the console consumer is working. Not sure what is the issue with CDK 4.0. CDK 3.1 brings in Kafka 1.0.1

kafka console consumer not receiving messages from console producer

2 Answers