case class implementation in spark

Question

I'm trying to implement case class in intellij idea, I'm getting below error. Can you please help me to fix the issue:

Code:

val conf = new SparkConf().setMaster("local").setAppName("case class")
val sc = new SparkContext(conf)

case class Employee (empno: String, ename: String, job: String, mgr: String,
            hiredate: String, sal: String, comm: String, deptno: String)

val empDF = sc
    .textFile("C:\\Spark\\emp.txt")
    .map(x=>x.split(","))
    .map(x => Employee(x(0), x(1), x(2), x(3), x(4), x(5), x(6), x(7)))

Error:

Exception in thread "main" org.apache.spark.SparkException: Task not serializable
    at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:403)
    at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:393)
    at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:162)
    at org.apache.spark.SparkContext.clean(SparkContext.scala:2326)
    at org.apache.spark.rdd.RDD.$anonfun$map$1(RDD.scala:371)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
    at org.apache.spark.rdd.RDD.map(RDD.scala:370)
    at SparkProject3$.main(SparkProject3.scala:20)
    at SparkProject3.main(SparkProject3.scala)

Caused by: java.io.NotSerializableException: SparkProject3$

Serialization stack:

Note: I am using spark 2.4.4 and scala 2.12.8 versions.

Thanks.

DNA DNA · Accepted Answer · 2019-12-17T08:49:19

Case classes have a reference to their enclosing class (if any).

So when Spark tries to serialize Employee it also tries to serialize the surrounding SparkProject3 class or object, but this fails because your driver class isn't serializable.

You need to define your Employee case class at the top level of your source file, outside the SparkProject3 class.

case class implementation in spark

1 Answers