Connector task state fails to connect

Question

Task state of a connector is getting failed with error:

org.apache.kafka.connect.errors.ConnectException: java.lang.NoClassDefFoundError

I am running kafka connect cluster in distributed mode and I am using kafka(0.10.0.2.5) connect of ambari deployment.

I gave debezium mysql connector path using export CLASSPATH=/path to connector/.

Loaded connector configuration into Kafka Connect using the following command:

curl -i -X POST -H "Accept:application/json" \ 
    -H  "Content-Type:application/json" http://localhost:8083/connectors/ \
    -d '{
      "name": "MYSQL_CONNECTOR",
      "config": {
            "connector.class": "io.debezium.connector.mysql.MySqlConnector",
            "database.hostname": "10.224.21.36",
            "database.port": "3306",
            "database.user": "root",
            "database.password": "shobhna",
            "database.server.id": "1",
            "database.server.name": "demo",
            "database.history.kafka.bootstrap.servers": "slnxhadoop04.noid.in:6669",
            "database.history.kafka.topic": "dbhistory.demo" ,
            "include.schema.changes": "true"
       }
    }'

Now after checking connector status, I am getting error:

 - {"name":"MYSQL_CONNECTOR","connector":{"state":"RUNNING","worker_id":"172.26.177.115:8083"},
   "tasks":[{"state":"FAILED","trace":"org.apache.kafka.connect.errors.ConnectException:
   java.lang.NoClassDefFoundError:
   org/apache/kafka/clients/admin/AdminClient\n\tat
   io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:218)\n\tat
   io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:45)\n\tat
   org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:137)\n\tat
   org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:140)\n\tat
   org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:175)\n\tat
   java.util.concurrent.Executors$RunnableAdapter.cal(Executors.java:511)\n\tat
   java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat
   java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat
   java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat
   java.lang.Thread.run(Thread.java:745)\nCaused by:
   java.lang.NoClassDefFoundError:
   org/apache/kafka/clients/admin/AdminClient\n\tat
   io.debezium.relational.history.KafkaDatabaseHistory.initializeStorage(KafkaDatabaseHistory.java:336)\n\tat
   io.debezium.connector.mysql.MySqlSchema.intializeHistoryStorage(MySqlSchema.java:260)\n\tat
   io.debezium.connector.mysql.MySqlTaskContext.initializeHistoryStorage(MySqlTaskContext.java:194)\n\tat
   io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:126)\n\t...
   9 more\nCaused by: java.lang.ClassNotFoundException:
   org.apache.kafka.clients.admin.AdminClient  \n\tat
   java.net.URLClassLoader.findClass(URLClassLoader.java:381)\n\tat
   java.lang.ClassLoader.loadClass(ClassLoader.java:424)\n\tat
   sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)\n\tat
   java.lang.ClassLoader.loadClass(ClassLoader.java:357)\n\t

Paste and include the error in your question instead of attaching a screenshot. — Giorgos Myrianthous

OneCricketeer OneCricketeer · Accepted Answer · 2018-08-28T01:17:09

It can't find a builtin Kafka class, not your Connector

NoClassDefFoundError: org/apache/kafka/clients/admin/AdminClient

...

i am using kafka(0.10.0.2.5)

Make sure you're running 1) a Connect Server version that matches your Kafka broker 2) using a Connector that uses code for that version of Connect

For example, AdminClient only exists in Kafka 0.11+.

In the recent HDP releases, you get Kafka 1.1 (different than 0.11), and this is the version that the latest Debezium is built and tested against https://debezium.io/docs/releases/

Debezium needs the AdminClient to make and register topic information, so I'm not sure if it'll work on old version such as 0.10

Its stated in the Kafka wiki that newer versions of Connect Server can communicate with old brokers, but the protocol used by the Connector classes is up for debate.

Connector task state fails to connect

1 Answers