scala worksheet throwing error with spark

Question

I am trying run spark tests using scala worksheet in Intellij. But at the line of instantiation of sparksession or sparkcontext, the worksheet is throwing following error

java.lang.ClassNotFoundException: org.apache.spark.serializer.JavaSerializer at java.net.URLClassLoader.findClass(scala tutorial.sc:377) at java.lang.ClassLoader.loadClass(scala tutorial.sc:420) at sun.misc.Launcher$AppClassLoader.loadClass(scala tutorial.sc:327) at java.lang.ClassLoader.loadClass(scala tutorial.sc:353) at java.lang.Class.forName0(scala tutorial.sc) at java.lang.Class.forName(scala tutorial.sc:344) at org.apache.spark.util.Utils$.classForName(scala tutorial.sc:225) at org.apache.spark.SparkEnv$.instantiateClass$1(scala tutorial.sc:256) at org.apache.spark.SparkEnv$.instantiateClassFromConf$1(scala tutorial.sc:277) at org.apache.spark.SparkEnv$.create(scala tutorial.sc:280) at org.apache.spark.SparkEnv$.createDriverEnv(scala tutorial.sc:170) at org.apache.spark.SparkContext.createSparkEnv(scala tutorial.sc:253) at org.apache.spark.SparkContext.(scala tutorial.sc:428) at org.apache.spark.SparkContext$.getOrCreate(scala tutorial.sc:2316) at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(scala tutorial.sc:827) at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(scala tutorial.sc:819) at #worksheet#.#worksheet#(scala tutorial.sc:116)

I have tried unchecking the Run worksheet in the compiler process under Worksheet option of Scala setting which doesn't throw the error but doesn't even compile.

It is a maven project and following are the dependencies I have

 <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>2.11.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>2.1.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.11</artifactId>
            <version>2.1.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>2.1.1</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/com.holdenkarau/spark-testing-base_2.10 -->
        <dependency>
            <groupId>com.holdenkarau</groupId>
            <artifactId>spark-testing-base_2.11</artifactId>
            <version>2.1.0_0.6.0</version>
        </dependency>

Can somebody help me resolve this issue?

Unchecking the Run worksheet in the compiler process worked for me with the Intellij 2017.2.15 — koiralo
Thanks @ShankarKoirala for the suggestion I tried that and didn't work 😞 — Ramesh Maharjan

Ramesh Maharjan Ramesh Maharjan · Accepted Answer · 2017-11-08T02:19:21

Somehow Use external compile server for scala option was unchecked which is under Scala Compile Server settings.

So in addition to following settings.

I had to enable the following setting

I hope this would be helpful for everyone having the same problem as me.

scala worksheet throwing error with spark

1 Answers