Kafka producer throws "Received unknown topic or partition error" when sending to topic created via AdminClient createTopics method

5
votes

I have issue with topics created using AdminClient createTopics. In my application I have following sequence:

  1. create new topic with 1 partition, replication factor set to 1 using AdminClient.createTopics
  2. wait onAdminClient.createTopics KafkaFuture result
  3. immediately send a new message to newly created topic (usually the time between operation 2 and 3 is about 200 milliseconds).

My code is as follows:

adminClient
  .createTopics(Collections.singleton(new NewTopic(targetTopic, 1, (short) 1)))
  .values()
  .get(targetTopic)
  .get();
producer.send(new ProducerRecord<>(targetTopic, data));

From time to time producer doesn't see the created topic and throws following exception:

[Producer clientId=producer-1] Error while fetching metadata with correlation id 5 : {targetTopic=UNKNOWN_TOPIC_OR_PARTITION}

[Producer clientId=producer-1] Received unknown topic or partition error in produce request on partition targetTopic. The topic/partition may not exist or the user may not have Describe access to it

This issue is very rare (< 0,1% all created topics).

Is it guaranteed that when AdminClient.createTopics Kafka future is completed then topic is created and Kafka producer should see that topic ? If not then which method of topic creation can give me such guarantee ?

I am using kafka-clients:2.0.0 and Kafka HD service on Azure. My cluster consists of 3 Zookeeper and 3 Kafka nodes.

1
javaapache-kafka

1 Answers

2
votes

Kafka client does not guarenteed about atomic topic creation. You can read it from this documentation. When you create a topic kafka will notify distributed systems about topics information so the action is running on different machines and these can't be atomic without distributed transactional management.