Kafka - Idempotent producer in "exactly once delivery" semantic

Question

From last version of Kafka (0.11.0.0) released the 28th of June 2017, the kafka team provided new features in order to support exactly once delivery. After I downloaded the latest version I tried configuring the Producer (executed through kafka-console-producer.sh script) as described in Producer configs: I set enable.idempotence=true and transactional.id=0A0A.

The problem is that when I start the producer I get a ConfigException saying that acks must be set to all or -1 (even if I set it in the producer.properties file which I pass as argument to the console script).

Could be the root cause that idempotence cannot be set using the console script? Moreover, is there a way to do an atomic transaction producing messages through the provided console script?

Details:

• Implementation details and motivations
• Exactly once semantics

In sythesis, the adopted solution is based upon two main concepts:

• idempotent Producer able to write a specific message only once, thanks to the introduction of transactional ids in producers config that guarantee atomicity property also in case of multiple partitions for a single topic);
• on the Consumer side, through the isolation.level=read_committed property, we are now able to read messages only after the transaction is committed.

Answer 1

Console producer sets its own defaults. Try adding --request-required-acks "all" or --request-required-acks -1 to set acks to all in place of the default of 1.