Kafka to Snowflake - Failed to find any class that implements Connector

Question

Attempting to connect to a Kafka cluster and write data to Snowflake from a topic.

The error I'm getting is:

java.util.concurrent.ExecutionException: org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches com.snowflake.kafka.connector.SnowflakeSinkConnector

available connectors are: PluginDesc{klass=class com.snowflake.kafka.connector.SnowflakeSinkConnector, name='com.snowflake.kafka.connector.SnowflakeSinkConnector', version='1.0.0', encodedVersion=1.0.0, type=sink, typeName='sink', location='file:path_to_kafka/kafka_2.12-2.3.0/libs/snowflakeinc-snowflake-kafka-connector/'}

The command I'm using to run (excluding the standard zookeeper / kafka start commands):

/path_to_kafka/kafka_2.12-2.3.0/bin/connect-standalone.sh /path_to_kafka/kafka_2.12-2.3.0/config/connect-standalone.properties /path_to_kafka/kafka_2.12-2.3.0/config/snowflake_properties.properties

What I've done so far:

Updated plugin.path in /path_to_kafka/kafka_2.12-2.3.0/config/connect-standalone.properties

with

plugin.path=/usr/share/java,/path_to_kafka/kafka_2.12-2.3.0/libs,/path_to_kafka/kafka_2.12-2.3.0/libs/snowflakeinc-snowflake-kafka-connector

I've researched similar cases online, but while many of those were users who received the same error message, their desired connector wasn't listed in the list of available connectors as it is for me.

I initially tried placing the snowflake jar directly in the libs folder, then again as the entire package structure, but neither work.

After each attempt, I've stopped zookeeper / kafka and restarted them, but still get the same error.

I've done tests with the the jar paths added to PATH, but I receive same failed result.

Where I've looked:

https://docs.confluent.io/3.1.1/connect/userguide.html#installing-connector-plugins

Kafka Connect can't find connector

Kafka Connect cant' find class of developed plugin

Kafka Connect failed to add MySqlConnector

https://github.com/DataReply/kafka-connect-mongodb/issues/23

https://github.com/confluentinc/kafka-connect-jdbc/issues/476

https://github.com/wepay/kafka-connect-bigquery/issues/109

How to copy or configure kafka connect plugin files?

Where I downloaded my connector: https://mvnrepository.com/artifact/com.snowflake/snowflake-kafka-connector/1.0.0

I've been on this for a few days now, and I feel like I've tried all of these suggestions. However, none of them work for me and I don't know what I'm doing wrong.

Answer 1

From the connector documentation:

Copy the JAR files you downloaded […] to the /libs folder.

To make it into the Kafka command's java runtime classpath, the jar(s) need to directly be under your /path_to_kafka/kafka_2.12-2.3.0/libs/ directory. From your description, they appear to be inside of a sub-directory (/path_to_kafka/kafka_2.12-2.3.0/libs/snowflakeinc-snowflake-kafka-connector/).

Answer 2

I was able to overcome this issue in the Debian "cp-kafka-connect" image.

While installing the connector plugins manually, I tried to:

1. Mount the connector plugin files through volumes and extract the plugins manually in the CONNECT_PLUGIN_PATH.
2. Run confluent-hub install (once the connect POD is spun up).
   

Both of the above options required a POD restart.

To overcome this manual POD restart, you can directly use the following command to install the connector plugin from within the image.

confluent-hub install snowflakeinc/snowflake-kafka-connector:1.4.3

Once the coonnector plugin in installed, you can install custom connectors:

curl -X POST -H "Content-Type: application/json" --data @ODX.json http://localhost:8083/connectors

Note: No other changes are required in CLASSPATH or CONNECT_PLUGIN_PATH.