Scala: Spark exception when project includes Akka

Question

I have a project that uses spark, and I want to use Akka in it. The project worked fine before, but when I added this to build.sbt:

libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.5.3"

And try to run the project, I get this error:

[error] (run-main-0) org.apache.spark.SparkException: Job aborted due to stage failure: Task serialization failed: java.lang.ClassNotFoundException: scala.Function0
[error] java.lang.ClassNotFoundException: scala.Function0
[error] at sbt.classpath.ClasspathFilter.loadClass(ClassLoaders.scala:63)
[error] at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
[error] at java.lang.Class.forName0(Native Method)
[error] at java.lang.Class.forName(Class.java:348)
[error] at com.twitter.chill.KryoBase$$anonfun$1.apply(KryoBase.scala:41)
[error] at com.twitter.chill.KryoBase$$anonfun$1.apply(KryoBase.scala:41)
[error] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
[error] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
[error] at scala.collection.immutable.Range.foreach(Range.scala:160)
[error] at
scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
[error] at scala.collection.AbstractTraversable.map(Traversable.scala:104)
[error] at com.twitter.chill.KryoBase.(KryoBase.scala:41)

Any ideas?

Dmytro Mitin Dmytro Mitin · Accepted Answer · 2017-09-10T16:33:36

With such build.sbt

scalaVersion := "2.11.11"
libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.5.3"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.2.0"

both HelloWorld in Akka and WordCount in Spark work fine. Akka can work with 2.10-2.12, but Spark with 2.10-2.11. It's hard to say more without code samples and build.sbt.

Scala: Spark exception when project includes Akka

1 Answers