Unable to set Python 3 as default version for Apache spark in MacOS

Question

I have been trying to keep Python 3 as default version for Spark in my Macbook, But its not setting up. I am unable to run my programs also due to that .

This is the error I am getting in Pycharm

"Exception: Python in worker has different version 2.7 than that in driver 3.7, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.

Can you please help me on this?

Answer 1

Just set the environment variable:

export PYSPARK_PYTHON=python3

in case you want this to be a permanent change add this line to pyspark script. If python3 is not accessible, you need to pass path to it instead.