Kafka Idempotent Producer becoming latency bottleneck

Question

We are using Idempotent producer mode in kafka producer because we want the data to be stored exactly once and gapless and in order at the broker. This kind of slows down as the max inflight requests to broker can only be 5. Also with 2 min.insync.replicas and idempotent producer I am seeing average latency on a 100k message burst to be about 20ms to 30ms on same machine where producer, consumer and kafka brokers are running.

Any suggestions on how latency can be improved while keeping the semantics "exactly once and gapless and in order". Also messages are financial trade/quote messages so we do not want any data loss or gaps.

Answer 1

For one of our telco use-case data volume was quite higher and message needed to be unique so we dumped data to DB with unique index on key column and used kafka-connect to read data from source and further business logic.

The advantage will be that you need not to write custom logic to analyze data before processing and it guarantees unique messages.