Apply a Quota to a Kafka Connect consumer group

Question

I have Kafka Connect JDBC sink connectors writing to various databases and I'd like to throttle the traffic to one database. The Kafka quotas feature can set a consumer_byte_rate quota for a client ID, but Kafka Connect client IDs look like consumer-1234 and are dynamically assigned to connectors. So if my sink connector is rebalanced, it will be assigned all new client IDs. I tried setting a quota using my sink connector consumer group ID as the client ID, but that doesn't work. Is there any way to set a quota for a Kafka Connect consumer group?

Answer 1

If you upgrade to Apache Kafka 2.3 you'll benefit from KIP-411: Make default Kafka Connect worker task client IDs distinct . You can see an example of it in action here. However, you'd have to test if the client-id is deterministic since quotas can't be wildcarded.