value lookup is not a member of org.apache.spark.rdd.RDD[(String, String)]

Question

I have got a problem when I tired to compile my scala program with SBT. I have import the class I need .Here is part of my code.

import java.io.File
import java.io.FileWriter
import java.io.PrintWriter
import java.io.IOException
import org.apache.spark.{SparkConf,SparkContext}
import org.apache.spark.rdd.PairRDDFunctions
import scala.util.Random    
......
val data=sc.textFile(path)
    val kv=data.map{s=>
      val a=s.split(",")
      (a(0),a(1))
    }.cache()
    kv.first()
    val start=System.currentTimeMillis()
    for(tg<-target){
      kv.lookup(tg.toString)
    }

The error detail is :

value lookup is not a member of org.apache.spark.rdd.RDD[(String, String)]
[error]           kv.lookup(tg.toString)

What confused me is I have import import org.apache.spark.rdd.PairRDDFunctions, but it doesn't work . And when I run this in Spark-shell ,it runs well.

maasg maasg · Accepted Answer · 2015-05-20T07:45:55

import org.apache.spark.SparkContext._ to have access to the implicits that let you use PairRDDFunctions on a RDD of type (K,V). There's no need to directly import PairRDDFunctions

value lookup is not a member of org.apache.spark.rdd.RDD[(String, String)]

1 Answers