Java Class not Found Exception while doing Spark-submit Scala using sbt

Question

Here is my code that i wrote in scala

package normalisation

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.spark.sql.SQLContext
import  org.apache.hadoop.fs.{FileSystem,Path}


object Seasonality {
 val amplitude_list_c1: Array[Nothing] = Array()
 val amplitude_list_c2: Array[Nothing] = Array()
 def main(args: Array[String]){
   val conf = new SparkConf().setAppName("Normalization")
   val sc = new SparkContext(conf)
   val sqlContext = new org.apache.spark.sql.SQLContext(sc)
   val line = "MP"
   val ps = "Test"
   val location = "hdfs://ipaddress/user/hdfs/{0}/ps/{1}/FS/2018-10-17".format(line,ps)
   val files = FileSystem.get(sc.hadoopConfiguration ).listStatus(new Path(location))
   for (each <- files) {
     var ps_data = sqlContext.read.json(each)

   }
   println(ps_data.show())
 }

The error I received when compiled using sbt package is hereimage

Here is my build.sbt file

name := "OV"

scalaVersion := "2.11.8"

// https://mvnrepository.com/artifact/org.apache.spark/spark-core libraryDependencies += "org.apache.spark" %% "spark-core" % "2.3.1"

// https://mvnrepository.com/artifact/org.apache.spark/spark-sql libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.3.1"

Dominic Egger Dominic Egger · Accepted Answer · 2018-11-14T10:57:41

in Spark Versions > 2 you should generally use SparkSession. See https://spark.apache.org/docs/2.3.1/api/scala/#org.apache.spark.sql.SparkSession

also then you should be able to do

val spark:SparkSession = ???
val location = "hdfs://ipaddress/user/hdfs/{0}/ps/{1}/FS/2018-10-17".format(line,ps)
spark.read.json(location)

to read all your json files in the directory.

Also I think you'd also get another compile error at

for (each <- files) {
  var ps_data = sqlContext.read.json(each)
}
println(ps_data.show())

for ps_data being out of scope.

If you need to use SparkContext for some reason it should indeed be in spark-core. Have you tried restarting your IDE, cleaned caches, etc?

EDIT: I just notices that build.sbt is probably not in the directory where you call sbt package from so sbt won't pick it up

Java Class not Found Exception while doing Spark-submit Scala using sbt

1 Answers