Scala code works on Spark Terminal but not in eclipse

0
votes

The code below works fine when I run it on Spark Terminal but in eclipse it throws an error. What might be the reason. Please let me know if you need more information.

val IABLabels= IAB.zip(labels)
val temp1 = IABLabels.groupBy(x=>x._2).mapValues( _.map( _._1 ))

Error in Eclipse:
value mapValues is not a member of org.apache.spark.rdd.RDD[(Int, Iterable[(String, Int)])]

The code runs perfectly fine on Spark shell.

1
eclipsescalaapache-spark

1 Answers

7
votes

You should use this import to access extra functions on RDD's of (key,value) pairs through an implicit conversion:

import org.apache.spark.SparkContext._

You can check the API docs for further details.