Too many UnknownProducerIdException in kafka broker for kafka internal topics created for kafka streams application

Question

One of Kafka stream application is generating a lot of Unknown Producer Id errors in the Kafka brokers as well as on the consumer side.

Stream Configs are as below:

final Properties streamsConfiguration = new Properties();
    streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, appName);
    streamsConfiguration.put(StreamsConfig.CLIENT_ID_CONFIG,appName + "-Client");
    streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, this.bootstrapServer);
    streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.Long().getClass().getName());
    streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
    streamsConfiguration.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG,StreamsConfig.EXACTLY_ONCE);
    streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, offset);
    streamsConfiguration.put(StreamsConfig.STATE_DIR_CONFIG,state_dir);
    streamsConfiguration.put(StreamsConfig.REPLICATION_FACTOR_CONFIG,defaultReplication);
    return streamsConfiguration;

Error on the broker side:

Error on the consumer side:

custom configuration for repartition internal topic: prod.Prod-Job-Summary-v0.4-KTABLE-AGGREGATE-STATE-STORE-0000000049-repartition

What can be the reason behind these?

Unknown Unknown · Accepted Answer · 2019-01-22T14:22:06

It's a known issue. See KAFKA-7190 Under low traffic conditions purging repartition topics cause WARN statements about UNKNOWN_PRODUCER_ID and KIP-360: Improve handling of unknown producer.

Too many UnknownProducerIdException in kafka broker for kafka internal topics created for kafka streams application

1 Answers