Kafka UNKNOWN_PRODUCER_ID exception

Question

I sometimes find UNKNOWN_PRODUCER_ID exception when using kafka streams.

2018-06-25 10:31:38.329  WARN 1 --- [-1-1_0-producer] o.a.k.clients.producer.internals.Sender  : [Producer clientId=default-groupz-7bd94946-3bc0-4400-8e73-7126b9b9c0d4-StreamThread-1-1_0-producer, transactionalId=default-groupz-1_0] Got error produce response with correlation id 1996 on topic-partition default-groupz-mplat-five-minute-stat-urlCount-counts-store-changelog-0, retrying (2147483646 attempts left). Error: UNKNOWN_PRODUCER_ID

Referred to official documents:

This exception is raised by the broker if it could not locate the producer metadata associated with the producerId in question. This could happen if, for instance, the producer's records were deleted because their retention time had elapsed. Once the last records of the producerId are removed, the producer's metadata is removed from the broker, and future appends by the producer will return this exception.

It says one possibility is that a producer is idle for more than retention time (by default a week) so the producer's metadata will be removed from broker. Are there any other reasons that brokers could not locate producer metadata?

Answer 1

You might be experiencing https://issues.apache.org/jira/browse/KAFKA-7190. As it says in that ticket:

When a streams application has little traffic, then it is possible that consumer purging would delete even the last message sent by a producer (i.e., all the messages sent by this producer have been consumed and committed), and as a result, the broker would delete that producer's ID. The next time when this producer tries to send, it will get this UNKNOWN_PRODUCER_ID error code, but in this case, this error is retriable: the producer would just get a new producer id and retries, and then this time it will succeed.

This issue is also being tracked at https://cwiki.apache.org/confluence/display/KAFKA/KIP-360%3A+Improve+handling+of+unknown+producer

Answer 2

Two reasons might delete your producer's metadata:

1. The log segments are deleted due to hitting retention time.
2. The producer state might get expired due to inactivity which is controlled by the setting transactional.id.expiration.ms which defaults to 7 days

So if your Kafka is < 2.4 you can workaround this by increasing the retention time(considering that your system allows that) of your topic's log(e.g 30 days) and to increase the transactional.id.expiration.ms setting( to 24 days) until KIP-360 is released:

log.retention.hours=720

transactional.id.expiration.ms=2073600000

This shall guarantee that for low-traffic topics(messages written rarely than 7 days), your producer's metadata state will remain stored in broker's memory for a longer period, thus decreasing the risk of getting UnknownProducerIdException.