Adding a connector to Kafka Connect

Question

I am using Confluent Kafka Docker image, specifically using this: https://github.com/confluentinc/cp-docker-images/tree/4.0.x/examples/cp-all-in-one

I want to add the MySQL connector, by:

downloading the version 1.5.46 of the connector (https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz)
mounting a volume with a jar that comes out of the downloaded connector (mysql-connector-java-5.1.46-bin.jar)
adding the CONNECT_PLUGIN_PATH to the docker compose file

Basically, extrapolating the part of interest form the compose file, this is the important snippet:

environment:
  <cut>
  CONNECT_PLUGIN_PATH: '/etc/kafka-connect/jars/'
volumes:
  - "./mysql_connector:/etc/kafka-connect/jars/"

The JAR is correctly present in the Docker container.

But, of course, if I try to use the connector, I can't find in the list of the enabled connectors. If I go to control center, I see:

[{
    "class": "org.apache.kafka.connect.file.FileStreamSinkConnector",
    "type": "sink",
    "version": "1.0.0-cp1"
}, {
    "class": "org.apache.kafka.connect.file.FileStreamSourceConnector",
    "type": "source",
    "version": "1.0.0-cp1"
}]

OneCricketeer OneCricketeer · Accepted Answer · 2018-03-18T18:43:01

Your plugin path needs to contain the path to the actual JDBC Connect JAR files as well as the MySQL JAR

If you mount the MySQL JAR under this path

/usr/share/java/kafka-connect-jdbc

Then set the CONNECT_PLUGIN_PATH=/usr/share/java, then you have access to all bundled connectors. And the JDBC Connector will be able to read/write MySQL data.

Adding a connector to Kafka Connect

1 Answers