Exception in thread “streaming-start” java.lang.NoClassDefFoundError: twitter4j/StreamListener

1
votes

I am trying to stream Twitter data into spark streaming. I have exported additional jars using following command -

spark-shell --jars twitter4j-core-4.0.1.jar,spark-streaming-twitter_2.11-1.6.2.jar

But at time of ssc.start() , getting following error -

scala> ssc.start()

Exception in thread "streaming-start" java.lang.NoClassDefFoundError: twitter4j/StreamListener at org.apache.spark.streaming.twitter.TwitterInputDStream.getReceiver(TwitterInputDStream.scala:55)

at org.apache.spark.streaming.scheduler.ReceiverTracker$$anonfun$8.apply(ReceiverTracker.scala:415)

at org.apache.spark.streaming.scheduler.ReceiverTracker$$anonfun$8.apply(ReceiverTracker.scala:414)

at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)

at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)

at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)

at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)

at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)

at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)

at org.apache.spark.streaming.scheduler.ReceiverTracker.launchReceivers(ReceiverTracker.scala:414)

at org.apache.spark.streaming.scheduler.ReceiverTracker.start(ReceiverTracker.scala:157)

at org.apache.spark.streaming.scheduler.JobScheduler.start(JobScheduler.scala:82)

at org.apache.spark.streaming.StreamingContext$$anonfun$liftedTree1$1$1.apply$mcV$sp(StreamingContext.scala:610)

at org.apache.spark.streaming.StreamingContext$$anonfun$liftedTree1$1$1.apply(StreamingContext.scala:606)

at org.apache.spark.streaming.StreamingContext$$anonfun$liftedTree1$1$1.apply(StreamingContext.scala:606)

at org.apache.spark.util.ThreadUtils$$anon$1.run(ThreadUtils.scala:122)

Caused by: java.lang.ClassNotFoundException: twitter4j.StreamListener

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

... 16 more

18/03/29 16:20:55 INFO StreamingContext: StreamingContext started

I am code given in following link-

https://acadgild.com/blog/streaming-twitter-data-using-spark/

Thanks..

1
apache-sparkspark-dataframespark-streaming

1 Answers

0
votes

It seems there is a missing dependency , that you need to add

please try to add this jar as a dependency , it has the class that you need

twitter4j.StreamListener

https://mvnrepository.com/artifact/org.twitter4j/twitter4j-stream/4.0.7

You will need to find the jar version which is build for your spark version and scala/java/python version