When I create a source or sink connector using Confluent Control Center where does it save the settings related to that connector? Are there files I can browse? We are planning to create 50+ connectors and at one point we need to copy them from one environment to another, I was wondering if there is an easy way to do that.
2 Answers
1
votes
Kafka Connect in distributed mode uses Kafka topics for storing configuration.
Kafka Connect supports a REST API. You can use this for viewing existing connector configuration, creating new ones (including programatically/automatically for 50+ new connectors), starting/stopping connectors, etc.
The REST API is documented here.
0
votes
Kafka Connect distributed mode is started with a property file. That property file defines a "config topic".
The connectors you're able to load, however, are not stored there - that's only for the running source/sink configurations.
The classes themselves are bundled as JAR files in the classpaths of the individual Connect Workers, and Control Center has no current way of provisioning new Connect classes. In other words, you must use something like Ansible or manually connect to each worker, download the Connect type you want, and extract it next to the other connects.
For example, let's pretend you wanted the Syslog connector.
You'd already have folders for these under usr/share/java in the Confluent installation
kafka-connect-hdfs
kafka-connect-jdbc
...
So, you download or build that Syslog connector, make a kafka-connect-syslog folder, and drop all necessary jar libraries there.
Once you do this for all connect instances, you'll need to also restart the Kafka Connect process on those machines.
Once Control Center connects back to the Connect server, you'll be able to configure your new Connect classes
