Spark Streaming using Apache Spark 2.2.1- java.lang.NoClassDefFoundError: org/apache/spark/Logging

Question

I have created a maven project in java eclipse, and added the dependencies in the pom.xml file.

<dependencies>
    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>2.10.2</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.10</artifactId>
        <version>2.2.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming_2.10</artifactId>
        <version>2.2.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming-twitter_2.10</artifactId>
        <version>1.3.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-examples_2.10</artifactId>
        <version>1.1.0</version>
    </dependency>   
</dependencies>

<properties>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>
</properties>

Then i created a java class and added the following codes to get data streams from a twitter application i previously created here

When i run the java application, I get the following error.

Exception in thread "main" java.lang.NoClassDefFoundError: 
org/apache/spark/Logging
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(Unknown Source)
at java.security.SecureClassLoader.defineClass(Unknown Source)
at java.net.URLClassLoader.defineClass(Unknown Source)
at java.net.URLClassLoader.access$100(Unknown Source)
at java.net.URLClassLoader$1.run(Unknown Source)
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.apache.spark.streaming.twitter.TwitterUtils$.createStream( 
TwitterUtils.scala:44)
at org.apache.spark.streaming.twitter.TwitterUtils$.createStream( 
TwitterUtils.scala:56)
at org.apache.spark.streaming.twitter.TwitterUtils.createStream( 
TwitterUtils.scala)
at streamingtest1.TweetStreamTest1.main(TweetStreamTest1.java:27)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 16 more

Could you please give a try by changing scala version to 2.11.8. — args
I'm getting another error Exception in thread "main" java.lang.NoClassDefFoundError: scala/xml/MetaData . — YamK

args args · Accepted Answer · 2018-02-26T14:53:50

spark UI uses scala package pls try to add the below dependency.

<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-xml</artifactId>
<version>2.11.0-M4</version>
</dependency>

Spark Streaming using Apache Spark 2.2.1- java.lang.NoClassDefFoundError: org/apache/spark/Logging

1 Answers