Kafka Streams Warning for unsubscribed topics

Question

In my kafka streams application, I have subscribed to a list of topics, lets say ( Topic1, Topic2, Topic3 ). Overall, broker has more than 3 topics(100s) and other stream applications are using those topics.

In my application ( which is consuming Topic1, Topic2 & Topic3), I keep on seeing the WARNing related to other topics also.

14:02:33.910 478517 [Application-468d913c-bca0-4ad9-baec-5b51ffa597d2-StreamThread-4] WARN  o.a.kafka.clients.NetworkClient - [Consumer clientId=Application-468d913c-bca0-4ad9-baec-5b51ffa597d2-StreamThread-4-consumer, groupId=Application] 1389 partitions have leader brokers without a matching listener, including [Topic4, Topic5, Topic6, Topic7, ..., Topic8-Link-Store-changelog-5 ]

the message is printed by following kafka code https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java

@Override
        public void handleSuccessfulResponse(RequestHeader requestHeader, long now, MetadataResponse response) {
            // If any partition has leader with missing listeners, log up to ten of these partitions
            // for diagnosing broker configuration issues.
            // This could be a transient issue if listeners were added dynamically to brokers.
            List<TopicPartition> missingListenerPartitions = response.topicMetadata().stream().flatMap(topicMetadata ->
                topicMetadata.partitionMetadata().stream()
                    .filter(partitionMetadata -> partitionMetadata.error() == Errors.LISTENER_NOT_FOUND)
                    .map(partitionMetadata -> new TopicPartition(topicMetadata.topic(), partitionMetadata.partition())))
                .collect(Collectors.toList());
            if (!missingListenerPartitions.isEmpty()) {
                int count = missingListenerPartitions.size();
                log.warn("{} partitions have leader brokers without a matching listener, including {}",
                        count, missingListenerPartitions.subList(0, Math.min(10, count)));
            }

Is this expected ? Why are these warning messages are posted in my application ?

OneCricketeer OneCricketeer · Accepted Answer · 2020-01-21T09:45:24

Note

This could be a transient issue if listeners were added dynamically to brokers

In any case, this would indicate an issue with the listeners property defined on the brokers.

For example, you've connected to PLAINTEXT://kafka:9092, but the server actually starts with PLAINTEXT_SASL://

Kafka Streams Warning for unsubscribed topics

1 Answers