Error when running pyspark

Question

I tried to run pyspark via terminal. From my terminal, I runs snotebook and it will automatically load jupiter. After that, when I select python3, the error comes out from the terminal.

[IPKernelApp] WARNING | Unknown error in handling PYTHONSTARTUP file /Users/simon/spark-1.6.0-bin-hadoop2.6/python/pyspark/shell.py

Here's my .bash_profile setting:

export PATH="/Users/simon/anaconda/bin:$PATH"

export SPARK_HOME=~/spark-1.6.0-bin-hadoop2.6

export PATH=$PATH:$SPARK_HOME/bin

export PYSPARK_DRIVER_PYTHON=jupyter

export PYSPARK_DRIVER_PYTHON_OPTS='notebook'

export PYSPARK_PYTHON=python3

alias snotebook='$SPARK_HOME/bin/pyspark'

Please let me know if you have any ideas, thanks.

Answer 1

You need to add below line in your code

PYSPARK_DRIVER_PYTHON=ipython

or

PYSPARK_DRIVER_PYTHON=ipython3

Hope it will help.

Answer 2

In my case, I was using a virtual environment and forgot to install Jupyter, so it was using some version that it found in the $PATH. Installing it inside the environment fixed this issue.

Answer 3

Spark now includes PySpark as part of the install, so remove the PySpark library unless you really need it.

Remove the old Spark, install latest version. Install (pip) findspark library.

In Jupiter, import and use findspark:

import findspark
findspark.init()

Quick PySpark / Python 3 Check

import findspark

findspark.init()

from pyspark import SparkContext 

sc = SparkContext()

print(sc)

sc.stop()