Apache Spark: java.lang.NoSuchMethodError .rddToPairRDDFunctions

Question

sbt package runs just fine, but after spark-submit I get the error:

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.SparkContext$.rddToPairRDDFunctions(Lorg/apache/spark/rdd/RDD;Lscala/reflect/ClassTag;Lscala/reflect/ClassTag;Lscala/math/Ordering;)Lorg/apache/spark/rdd/PairRDDFunctions; at SmokeStack$.main(SmokeStack.scala:46) at SmokeStack.main(SmokeStack.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Here is the offending line:

val sigCounts = rowData.map(row => (row("Signature"), 1)).countByKey()

rowData is an RDD Map[String, String]. "Signature" key exists in all items in the map.

I suspect this may be a build issue. Below is my sbt file:

name := "Example1"
version := "0.1"
scalaVersion := "2.11.8"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.2.0"
scalacOptions ++= Seq("-feature")

I'm new to Scala so maybe the imports are not correct? I have:

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import scala.io.Source

That's fairly old Spark version. Try to use a bit newer one. And as mentioned above - your runtime and compile time versions of spark do not match. — JiriS
@maasg I changed compile and runtime Spark versions to match and everything works. Can you put this as the answer so I can accept it? — Daniel Kats

maasg maasg · Accepted Answer · 2016-11-02T22:38:47

java.lang.NoSuchMethodError is often an indication that the version the code was compiled against is on a higher version than the libraries used at runtime.

With Spark, that means that the Spark version used to compile is different from the one deployed (on the machine or cluster).

Aligning the versions between development and runtime should solve this issue.

Apache Spark: java.lang.NoSuchMethodError .rddToPairRDDFunctions

3 Answers