Topic can't be found when producing messages: UNKNOWN_TOPIC_OR_PARTITION

13
votes

I have a two-nodes Kafka cluster (EC2 instances) where each node is used as a separate broker. When I run a producer on the leader instance with the following command:

kafka-console-producer.sh   --broker-list localhost:9092 --topic test

I get the following errors.

test message [2017-01-09 13:22:39,483] WARN Error while fetching metadata with correlation id 0 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-01-09 13:22:39,562] WARN Error while fetching metadata with correlation id 1 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-01-09 13:22:39,663] WARN Error while fetching metadata with correlation id 2 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) ...

Listing the topics with kafka-topics.sh shows that the topic exists.

The description of the topic with:

kafka-topics.sh --zookeeper localhost:2181 --describe --topic test

returns

 Topic:test PartitionCount:8    ReplicationFactor:1 Configs:
     Topic: test    Partition: 0    Leader: 1   Replicas: 1 Isr: 1
     Topic: test    Partition: 1    Leader: 2   Replicas: 2 Isr: 2
     Topic: test    Partition: 2    Leader: 1   Replicas: 1 Isr: 1
     Topic: test    Partition: 3    Leader: 2   Replicas: 2 Isr: 2
     Topic: test    Partition: 4    Leader: 1   Replicas: 1 Isr: 1
     Topic: test    Partition: 5    Leader: 2   Replicas: 2 Isr: 2
     Topic: test    Partition: 6    Leader: 1   Replicas: 1 Isr: 1
     Topic: test    Partition: 7    Leader: 2   Replicas: 2 Isr: 2

I am using Kafka 0.10.1.1.

server.properties file contains:

listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://0.0.0.0:9092
port=9092
host.name=kafka-node1(kafka-node1 for the second host)
advertised.host.name=kafka-node1(kafka-node2 for the second host)
advertised.port=9092

When I try to produce messages from the second host I get the message below:

WARN Got error produce response with correlation id 1 on topic-partition test-4, retrying (2 attempts left). Error: NOT_LEADER_FOR_PARTITION (org.apache.kafka.clients.producer.internals.Sender) ....

Can anyone please help?

3
apache-kafkaconsumerproducer
Try to specify all brokers in --broker-list. This error may appear when producer tries to send a message to a partition that is not served by a broker. AFAIK, advertised.* settings are deprecated. – Yuri Tceretian
When I put both hosts in --broker-list I still get the same error. – saloua
You don't need port, host.name, advertised.host.name and advertised.port settings, just use listers and advertised.listeners only if clients outside the machine see an address different than the address where service is listening on. Then, for your setup try using broker-list equal to private IP of the machine where broker is running. – Luciano Afranllie
This error message also happen if sentry is active and the user has not the correct permissions. – ozw1z5rd
how about kafka-console-producer.sh --broker-list kafka-node1:9092 --topic test ? – Svend

3 Answers

1
votes

If you are getting UNKNOWN_TOPIC_OR_PARTITION then add the below properties to server.properties file :

listeners=PLAINTEXT://host.name:port
advertised.listeners=PLAINTEXT://host.name:port

in my cases values were as follows:

listeners=PLAINTEXT://localhost:9092
advertised.listeners=PLAINTEXT://localhost:9092
0
votes

Your topic is marked for deletion:

KAFKA_OPTS="-Djava.security.auth.login.config=/home/kafka/kafka_2.11-1.0.0/config/jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf" \
    bin/kafka-topics.sh --describe \
    --zookeeper zookeeper-server-01.eigenroute.com:2181,zookeeper-server-02.eigenroute.com:2181,zookeeper-server-03.eigenroute.com:2181/apps/kafka-cluster-demo
Topic:another-test-topic    PartitionCount:9    ReplicationFactor:2 Configs:    MarkedForDeletion:true
    Topic: another-test-topic   Partition: 0    Leader: none    Replicas: 2,1   Isr:
    Topic: another-test-topic   Partition: 1    Leader: none    Replicas: 1,2   Isr:
    Topic: another-test-topic   Partition: 2    Leader: none    Replicas: 2,1   Isr:
...
-4
votes

Make the replication factor 2 for topic test and you won't get this exception anymore. Since you have a 2 broker cluster, execute the command below to produce messages:

kafka-console-producer.sh   --broker-list localhost:9092,localhost:9093 --topic test

This will send messages to both of the brokers in the clusters.