Py4JJavaError when testing Pyspark in Jupyter notebook on a single machine

Question

I am new to Spark and recently installed it on a mac (with Python 2.7 in the system) using homebrew:

brew install apache-spark

and then installed Pyspark using pip3 in my virtual environment where I have python 3.6 installed.

/Users/xxx/venv/bin/python /Users/xxx/venv/bin/pip3 install pyspark

When I ran the code below in my Jupyter Notebook to test if Spark functions on a single machine:

from pyspark.context import SparkContext
sc = SparkContext.getOrCreate()

import random
num_samples = 100000000

def inside(p):     
    x, y = random.random(), random.random()
    return x*x + y*y < 1

count = sc.parallelize(range(0, num_samples)).filter(inside).count()

pi = 4 * count / num_samples
print(pi)

sc.stop()

I ran into the following error with sc.parallelize:

Py4JJavaError                             Traceback (most recent call last)
<ipython-input-3-482026ac7386> in <module>
      8     return x*x + y*y < 1
      9 
---> 10 count = sc.parallelize(range(0, num_samples)).filter(inside).count()
     11 
     12 pi = 4 * count / num_samples
~/venv/deep_learning/lib/python3.6/site-packages/pyspark/rdd.py in count(self)
   1139         3
   1140         """
-> 1141         return self.mapPartitions(lambda i: [sum(1 for _ in i)]).sum()
   1142 
   1143     def stats(self):

~/venv/deep_learning/lib/python3.6/site-packages/pyspark/rdd.py in sum(self)
   1130         6.0
   1131         """
-> 1132         return self.mapPartitions(lambda x: [sum(x)]).fold(0, operator.add)
   1133 
   1134     def count(self):

~/venv/deep_learning/lib/python3.6/site-packages/pyspark/rdd.py in fold(self, zeroValue, op)
   1001         # zeroValue provided to each partition is unique from the one provided
   1002         # to the final reduce call
-> 1003         vals = self.mapPartitions(func).collect()
   1004         return reduce(op, vals, zeroValue)
   1005 

~/venv/deep_learning/lib/python3.6/site-packages/pyspark/rdd.py in collect(self)
    887         
    888         with SCCallSiteSync(self.context) as css:
-> 889             sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
    890         return list(_load_from_socket(sock_info, self._jrdd_deserializer))
    891 

~/venv/deep_learning/lib/python3.6/site-packages/py4j/java_gateway.py in __call__(self, *args)
   1303         answer = self.gateway_client.send_command(command)
   1304         return_value = get_return_value(
-> 1305             answer, self.gateway_client, self.target_id, self.name)
   1306 
   1307         for temp_arg in temp_args:

~/venv/deep_learning/lib/python3.6/site-packages/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    326                 raise Py4JJavaError(
    327                     "An error occurred while calling {0}{1}{2}.\n".
-> 328                     format(target_id, ".", name), value)
    329             else:
    330                 raise Py4JError(

Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 0.0 failed 1 times, most recent failure: Lost task 2.0 in stage 0.0 (TID 2, 192.168.0.15, executor driver): org.apache.spark.api.python.PythonException: Traceback (most recent call last):

  File "/Users/xxx/venv/deep_learning/lib/python3.6/site-packages/pyspark/python/lib/pyspark.zip/pyspark/worker.py", line 477, in main
    ("%d.%d" % sys.version_info[:2], version))

Exception: Python in worker has different version 2.7 than that in driver 3.6, PySpark cannot run with different minor versions. Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.

I have configured the Pyspark path as follows in the /.bash_profile:

export SPARK_HOME=/usr/local/Cellar/apache-spark/3.0.1/libexec
export PYTHONPATH=/usr/local/Cellar/apache-spark/3.0.1/libexec/python/:$PYTHONP$
export PYSPARK_PYTHON=/Users/xxx/venv/bin/python
export PYSPARK_DRIVER_PYTHON=/Users/xxx/venv/bin/python

Does anyone know what I did wrong here? Any advice will be greatly appreciated.

I had a similar problem, I have noticed that I have multiple running instance of Spark and it gave me error for that. I have closed all instances and run my code again an everything works perfectly! — pari

renjie ge renjie ge · Accepted Answer · 2020-12-21T03:00:49

It seems this problem is specifically related to Pyspark. The problem can be solved by using the findspark package. Below is the quote from the findspark readme file:

PySpark isn't on sys.path by default, but that doesn't mean it can't be used as a regular library. You can address this by either symlinking pyspark into your site-packages, or adding pyspark to sys.path at runtime. findspark does the latter.

Adding the code below before initiating SparkContext solves the problem:

import findspark
findspark.init()

Py4JJavaError when testing Pyspark in Jupyter notebook on a single machine

1 Answers