Spark : Error Not found value SC

11
votes

I have just started with Spark. I have CDH5 Installed with Spark . However when I try to use sparkcontext it gives Error as below

<console>:17: error: not found: value sc
       val distdata = sc.parallelize(data)

I have researched about this and found error: not found: value sc

and tried to start spark context with ./spark-shell . It gives error No such File or Directory

5
apache-sparkapache-spark-sql

5 Answers

5
votes

You can either start spark-shell starting with ./ if you're in its exact directory or path/to/spark-shell if you're elsewhere.

Also, if you're running a script with spark-submit, you need to initialize sc as SparkContext first:

import org.apache.spark.SparkContext
import org.apache.spark.SparkConf

val conf = new SparkConf().setAppName("Simple Application")
val sc = new SparkContext(conf)
4
votes

There is another stackoverflow post that answers this question by getting sc(spark context) from spark session. I do it this way:

val spark = SparkSession.builder().appName("app_name").enableHiveSupport().getOrCreate()

val sc = spark.sparkContext

original answer here: Retrieve SparkContext from SparkSession

3
votes

Add spark directory to path then you may use spark-shell from anywhere.

Add import org.apache.spark.SparkContext if you are using it in a spark-submit job to create a spark context using:

val sc = new SparkContext(conf)

where conf is already defined.

0
votes

Starting a new terminal fixes the problem in my case.

0
votes

You need to run Hadoop daemons first (run this command "start-all.sh"). Then try