Kafka Connect - Delete Connector with configs?

Question

I know how to delete Kafka connector as mentioned here Kafka Connect - How to delete a connector

But I am not sure if it also delete/erase specific connector related configs, offsets and status from *.sorage.topic for that worker?

For e.g: Lets say I delete a connector having connector-name as"connector-abc-1.0.0" and Kafka connect worker was started with following config.

offset.storage.topic=<topic.name>.internal.offsets
config.storage.topic=<topic.name>.internal.configs
status.storage.topic=<topic.name>.internal.status

Now after DELETE call for that connector, will it erased all records from above internal topics for that specific connector?
So that I can create new connector with "same name" on same worker but different config(different offset.start or connector.class)?

OneCricketeer OneCricketeer · Accepted Answer · 2019-01-16T04:00:48

Since Kafka is append only, then only way the messages in those Conncet topics would be removed is if it were published with the connector name as the message key, and null as the value.

You could inspect those topics using console consumer to see what data is in them including --property print.key=true, and keep the consumer running when you delete a connector.

You can PUT a new config at /connectors/{name}/config, but any specific offsets that are used are dependent upon the actual connector type (sink / source); for example, there is the internal Kafka __consumer_offsets topic as well as the offset.storage.topic. I'm not sure changing connector.class would be a good idea with that in mind

Kafka Connect - Delete Connector with configs?

2 Answers