How can I set up an idempotent producer in Spring Cloud Stream using Kafka?

Question

I would like to take advantage of the idempotent producers introduced in Kafka 0.11. According to this Confluent blog post a new property was added to support this:

Idempotence: Exactly-once in order semantics per partition

To turn on this feature and get exactly-once semantics per partition—meaning no duplicates, no data loss, and in-order semantics—configure your producer to set “enable.idempotence=true”.

The point is neither Spring Cloud Stream nor Spring Kafka document the use of that property. How can we configure it in Spring Cloud Stream applications?

Answer 1

Arbitrary kafka properties can be set using spring.cloud.stream.kafka.bindings.<channel>.producer.configuration, as documented here.

So, simply set the property there.

Of course, you will need to override the default kafka binder to use the kafka11 artifact as discussed in the Ditmars release train release notes.

The kafka11 binder also supports transactions.