Kafka Connect cant' find class of developed plugin

Question

I created a plugin for kafka connect that implements SinkConnector, I packaged it into a jar using gradle jar task:

jar {
  archiveName='name.jar'
}

I copy it into the kafka cluster in a folder, and I set CLASSPATH=where my jar is. Then I execute the kafka script to start the standalone connect and it gives me an error saying that my class can't be found:

[2017-07-25 05:15:52,084] WARN could not get type for name mypackage.SplunkSinkConnector from any class loader (org.reflections.Reflections:384)
org.reflections.ReflectionsException: could not get type for name mypackage.SplunkSinkConnector
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:378)
        at org.reflections.ReflectionUtils.forNames(ReflectionUtils.java:397)
        at org.reflections.Reflections.getSubTypesOf(Reflections.java:367)
        at org.apache.kafka.connect.runtime.PluginDiscovery.connectorPlugins(PluginDiscovery.java:76)
        at org.apache.kafka.connect.runtime.PluginDiscovery.scanClasspathForPlugins(PluginDiscovery.java:70)
        at org.apache.kafka.connect.runtime.AbstractHerder$1.run(AbstractHerder.java:354)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: mypackage.SplunkSinkConnector
        at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:376)
        ... 6 more
[2017-07-25 05:15:52,419] INFO Reflections took 1586 ms to scan 62 urls, producing 3002 keys and 15379 values  (org.reflections.Reflections:229)
[2017-07-25 05:15:52,420] WARN could not get type for name mypackage.SplunkSinkConnector from any class loader (org.reflections.Reflections:384)
org.reflections.ReflectionsException: could not get type for name mypackage.SplunkSinkConnector
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:378)
        at org.reflections.ReflectionUtils.forNames(ReflectionUtils.java:397)
        at org.reflections.Reflections.getSubTypesOf(Reflections.java:367)
        at org.apache.kafka.connect.runtime.ConnectorFactory.getConnectorClass(ConnectorFactory.java:69)
        at org.apache.kafka.connect.runtime.ConnectorFactory.newConnector(ConnectorFactory.java:38)
        at org.apache.kafka.connect.runtime.AbstractHerder.getConnector(AbstractHerder.java:334)
        at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:233)
        at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:159)
        at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:93)
Caused by: java.lang.ClassNotFoundException: mypackage.SplunkSinkConnector
        at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:376)
        ... 8 more

Any idea why is not picking up my jar?

Thank you

================================

EDIT: Kakfa Connect version 10.2.1, as per script, classpath is calculated with: CLASSPATH="$CLASSPATH":"$KAFKA_HOME/libs/*"

Please add Kafka Connect version number information to help debug as CLASSPATH has changed depending on version? — Hans Jespersen
Kakfa Connect version 10.2.1, as per script, classpath is calculated with: CLASSPATH="$CLASSPATH":"$KAFKA_HOME/libs/*" — frm
I'm using a custom plugin with Kafka Connect and loading by CLASSPATH environment variable and it works fine for me. — clay

clay clay · Accepted Answer · 2017-07-26T16:32:46

Can you check your .jar file to make sure that class is there. Using Scala as a JVM shell:

// Or Maybe try $CLASSPATH with the full classpath that you are using.
scala -classpath path-to-jar.jar
// If class is not loaded, this will trigger an error.
classOf[mypackage.SplunkSinkConnector]

FYI, I am doing exactly this, using a custom .jar plugin with Kafka Connect loaded via the CLASSPATH environment variable.

UPDATE: Here is my build.gradle file for my plugin. IMO, this is the simplest way to build a Java .jar with some simple dependencies. I build the jar with gradle jar and it will be created at ./build/libs/(project-name).jar:

apply plugin: 'java'
apply plugin: 'idea'

sourceCompatibility = 1.8

repositories {
    mavenLocal()
    mavenCentral()
    maven { url "http://packages.confluent.io/maven/" }
}

configurations {
    all*.exclude group: 'org.slf4j', module: 'slf4j-log4j12'
    all*.exclude group: 'log4j'
}

dependencies {
    compile 'io.confluent:kafka-connect-storage-partitioner:3.2.1'
    compile 'org.apache.kafka:connect-api:0.10.2.1'

    testCompile 'junit:junit:4.+'
}

idea {
    project {
        languageLevel = '1.8'
    }
}

Kafka Connect cant' find class of developed plugin

1 Answers