Apache Kafka-Exception : org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/List;)

0
votes

When I submit Spark Structured Streaming application, I am getting below error:

java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/Collection;)

Spark version: 2.1.0 Using Scala version 2.11.1, OpenJDK 64-Bit Server VM, 1.8.0_151

Kafka Version: kafka_2.12-0.10.2.1 (Scala version 2.12, Kafka Version 0.10.2.1)

I have included spark-sql-kafka-0-10_2.11-2.1.0.jar & kafka-clients-0.10.2.1.jar files to spark.driver.extraClassPath & spark.executor.extraClassPath

Can anyone help me to find out why I am getting error?

Is it because different Scala version on Spark 2.1.0 (with Scala version 2.11.1) and Kafka 0.10.2.1 (with Scala version 2.12)? if it is then how to resolve it without installing Kafka 0.10.2.1 (with Scala version 2.11)?

1
apache-sparkapache-kafka
Any suggestions? – sam

1 Answers

1
votes

Include spark-streaming-kafka-0-10_2.11-2.1.0.jar in spark.driver.extraClassPath in spark-submit should solve your problem. Mostly, it is not related to differing scala versions.

Ex:

spark-submit \
      --conf "spark.driver.extraClassPath=/jar/path/spark-streaming-kafka-0-10_2.11-2.2.0.jar" \