scala code throw exception in spark

Question

I am new to scala and spark. Today I tried to write some code, and let it run on spark, but got an exception.

this code work in local scala

import org.apache.commons.lang.time.StopWatch
import org.apache.spark.{SparkConf, SparkContext}

import scala.collection.mutable.ListBuffer
import scala.util.Random

  def test(): List[Int] = {
    val size = 100
    val range = 100
    var listBuffer = new ListBuffer[Int] // here throw an exception
    val random = new Random()
    for (i <- 1 to size)
      listBuffer += random.nextInt(range)
    listBuffer.foreach(x => println(x))
    listBuffer.toList
  }

but when I put this code into spark, it throw an exception says:

15/01/01 14:06:17 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
Exception in thread "main" java.lang.NoSuchMethodError: scala.runtime.ObjectRef.create(Ljava/lang/Object;)Lscala/runtime/ObjectRef;
    at com.tudou.sortedspark.Sort$.test(Sort.scala:35)
    at com.tudou.sortedspark.Sort$.sort(Sort.scala:23)
    at com.tudou.sortedspark.Sort$.main(Sort.scala:14)
    at com.tudou.sortedspark.Sort.main(Sort.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

if I comment out the below code, the code work in spark

for (i <- 1 to size)

can someone explain why, please.

What is line 35 of Sort.scala? The new ListBuffer line? It seems odd that that throws an exception which goes away when you remove the for later. Please post a complete example that fails with spark — The Archetypal Paul
This looks like a scala version mismatch. Check that the version of scala you're using matches that that spark was built against, and the one on the cluster where you're running. — lmm

Chan Chan · Accepted Answer · 2015-01-04T06:15:06

Thanks @Imm, I have solved this issue. The root cause is that my local scala is 2.11.4, but my spark cluster is running at 1.2.0 version. The 1.2 version of spark was compiled by 2.10 scala.

So the solution is compile local code by 2.10 scala, and upload the compiled jar into spark. Everything works fine.

scala code throw exception in spark

1 Answers