This is our development environment
1) kafka cluster - version is 0.10 2) Spark cluster - 1.6 which has 0.9 Kafka jars
We are trying to produce() and consume() in spark cluster mode. (via spark-submit)
While running spark-submit job, spark chooses 0.9 version of kafka. The following is our observation
1) Producer – works fine ( 0.9 api and 0.10 api producer is compatible ) 2) Streaming Kafka Consumer using KafkaUtils – works fine ( seems here also 0.9 api and 0.10 api producer is compatible) 3) Consumer using subscribe() API – Errors out with the following message. Can someone help us know why is it failing ?
16/10/24 02:31:08 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/Collection;)V java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/Collection;)V at com.common.kafka.init(Kafkafunction.java:150) at com.client.Client.main(Client.java:100) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542) 16/10/24 02:31:08 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/Collection;)V)
