code using the Spark on pyspark

1
votes

I have problem in pyspark. I have executed just these lines on Spark using pyspark as given below, it gives me following error,

>>> text_file=sc.textFile("/Users/Aroon/spark/Scala/word")

Traceback (most recent call last): File "", line 1, in NameError: name 'sc' is not defined.

1
pythonapache-sparkpyspark
The Error says 'sc' is not defined which is quite self-explanatory. Perhaps you could show us more of your code so that we can help you. - Kenny Chan

1 Answers

2
votes

The error message quite clearly states that the sparkContext is not created on the sc variable. This would have automatically got create incase you are on the pyspark shell. Since it does not, I predict you are running it through spark-submit. In that case, add the below boiler plate code to get your script running.

from pyspark import SparkContext

sc = SparkContext(appName="Scalaword")

More information on what you have got would help you better.