Retrieve SparkContext from SparkSession

27
votes

I'm running a spark batch job and uses SparkSession as I need a lot of spark-sql features to process in each of my components. The SparkContext is initialized in my parent component and been passed to the child components as SparkSession.

In one of my child components, I wanted to add two more configurations to my SparkContext. Hence, I need to retrieve the SparkContext from the SparkSession, stop it and recreate the SparkSession with the additional configuration. To do so, how can I retrieve SparkContext from SparkSession?

2
scalaapache-spark
Isn't the SparkContext a child of SparkSession (spark.sparkContext)? - ayplam
@ayplam do you mean the sparkContext() method in SparkSession? There is no clear document on its usage. When I try using it in my method, it throws reference error. I'm using spark 2.2.0. - Naveen Balasubramanian
What if you try accessing it without parenthesis? I'm on spark 2.2.0 as well and I can retrieve the sparkcontext fine that way via spark-shell - ayplam
That works!! var spCon = sparkSession.sparkContext The documentation kinda mislead me. Thanks a lot!! - Naveen Balasubramanian

2 Answers

52
votes

Just to post as an answer - the SparkContext can be accessed from SparkSession using spark.sparkContext (no parenthesis)

3
votes

The sparkContext field does not seem to be public anymore(I am using Spark 2.3.2), however, you can retreive it using a method of the same name:

spark.sparkContext()

This is applicable to Spark Java only.