Kafka connect confluent elasticsearch sink (no class found error)

3
votes

I am very new to Kafka connect. I want to push my messages from Kafka topic to elasticsearch. After following the available documentation.. I downloaded and compiled elastic search sink from release tar.zip (https://github.com/confluentinc/kafka-connect-elasticsearch/releases)

I added the elastic search properties file and included the above jar in classpath. When I run kafka connect in standalone mode I get this error

./usr/bin/connect-standalone etc/schema-registry/connect-avro-standalone.properties etc/kafka-connect-elasticsearch/quickstart-elasticsearch.properties

[2016-11-13 00:05:38,768] ERROR Task elasticsearch-sink-0 threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:142) java.lang.NoClassDefFoundError:io/searchbox/client/JestClientFactory
at io.confluent.connect.elasticsearch.ElasticsearchSinkTask.start(ElasticsearchSinkTask.java:81)
at io.confluent.connect.elasticsearch.ElasticsearchSinkTask.start(ElasticsearchSinkTask.java:52)
at org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:207)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:139)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:140)
at o

NoClassDefFoundError:io/searchbox/client/JestClientFactory

I checkedthe pom.xml and it has Jest client dependency defined correctly. Am I missing anything?

Any pointers would be really appreciated.

Thanks, Rajesh

2
javaconfluent-platformapache-kafka-connect
Seems like something isn't compiled correctly or you haven't added all of the dependencies to the classpath? Maybe you can show what you've added to the classpath and make sure everything in the target directory is added to the classpath for the worker. In most cases, it's not going to be just one jar you need, but several. – dawsaw
I am just using export CLASSPATH=/vagrant/kafka-connect-elasticsearch-3.1.0/target/kafka-connect-elasticsearch-3.1.0.jar. Is there anything else I need to add to classpath? – Rajesh
Add everything in that target directory and give it a try. The other jars in the target directory contain dependencies for the connector as there's currently no uber jar packaging. – dawsaw
I added export CLASSPATH=/vagrant/kafka-connect-elasticsearch-3.1.0/target/kafka-connect-elasticsearch-3.1.0.jar:/vagrant/kafka-connect-elasticsearch-3.1.0/target/kafka-connect-elasticsearch-3.1.0-development/share/java/kafka-connect-elasticsearch/* and it worked – Rajesh
Appreciate your help @dawsaw – Rajesh

2 Answers

1
votes

It does seem like one of the dependencies is missing from the classpath. The packages on the Github Releases page do not include dependencies.

I would suggest to use the Confluent Open Source distribution and follow the quickstart.

-1
votes

You can also build fat jar. Add descriptorRefs

<build>
  <plugins>
  ....
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-assembly-plugin</artifactId>
      <configuration>
      ....
        <descriptorRefs>
          <descriptorRef>jar-with-dependencies</descriptorRef>
        </descriptorRefs>
      </configuration>
      ....
    </plugin>
  </plugins>
  ....
</build>

to pom, build and copy jar with dependencies to libs directory of Kafka.