ClassNotFoundException in SparkStreaming Example

Question

I am new to Spark streaming and trying to run a example from this tutorial and I am following MAKING AND RUNNING OUR OWN NETWORKWORDCOUNT. I have completed 8th step and made a jar from sbt.

Now I am trying to run deploy my jar using the command in 9th step like this:

bin/spark-submit --class "NetworkWordCount" --master spark://abc:7077 target/scala-2.11/networkcount_2.11-1.0.jar localhost 9999

but when I run this command I get following exception:

java.lang.ClassNotFoundException: NetworkWordCount at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.spark.util.Utils$.classForName(Utils.scala:229) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:700) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

jar that I have created contains "NetworkWordCount" class having the following code from the spark examples

package src.main.scala

import org.apache.spark.SparkConf
import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming.{Seconds, StreamingContext}

object NetworkWordCount {
  def main(args: Array[String]) {
    if (args.length < 2) {
      System.err.println("Usage: NetworkWordCount <hostname> <port>")
      System.exit(1)
    }

    //StreamingExamples.setStreamingLogLevels()

    // Create the context with a 1 second batch size
    val sparkConf = new SparkConf().setAppName("MyNetworkWordCount")
    val ssc = new StreamingContext(sparkConf, Seconds(1))

    val lines = ssc.socketTextStream(args(0), args(1).toInt, StorageLevel.MEMORY_AND_DISK_SER)
    val words = lines.flatMap(_.split(" "))
    val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)
    wordCounts.print()
    ssc.start()
    ssc.awaitTermination()
  }
}

I am unable to identify what am I doing wrong.

I think the exception says what the issue is: Warning: Local jar /usr/local/spark/target/scala-2.11/networkcount_2.11-1.0.jar does not exist, check that out. — maasg
made target/scala-2.11 directory and placed my jar there, now the warning has vanished but the exception is still there — M.Ahsen Taqi

maasg maasg · Accepted Answer · 2017-07-31T13:08:12

The spark-submit parameter --class takes a fully qualified class name. In the case of the code above, it should be src.main.scala.NetworkCount

bin/spark-submit --class src.main.scala.NetworkCount --master spark://abc:7077 target/scala-2.11/networkcount_2.11-1.0.jar localhost 9999

Note: the package name used looks like an IDE setup issue. src/main/scala is the typical root for a scala code base, and not a package name.

ClassNotFoundException in SparkStreaming Example

2 Answers