Kafka Streams Exactly Once Consumer Groups

Question

When I turn on exactly once processing I get the following error. NOTE: Our application are very secure and we only give kafka users and consumers access to resources that they explicitly need.

2019-04-22 15:28:09 INFO  (kafka.authorizer.logger)233 - Principal = User:xxx is Denied Operation = Describe from hos
xxx.xxx.xxx.xxx on resource = TransactionalId:application_consumer-0_16

With exactly once processing does kafka streams use a consumer group per stream task instead of a consumer group across all stream tasks?

Matthias J. Sax Matthias J. Sax · Accepted Answer · 2019-04-23T05:59:14

With exactly-once enabled, there is still only one consumer group that is the same as the application.id. However, instead of using one Producer per thread, one producer per task is used.

What you need is permission for transaction. The TransactionsId the error reports is from the producer of task 0_16. Each producer uses its own transactional ID, that is constructed as <application.id>-<taskId>.

For details, compare the docs: https://docs.confluent.io/current/kafka/authorization.html#using-acls

Kafka Streams Exactly Once Consumer Groups

1 Answers