pyspark interpreter not found in apache zeppelin

4
votes

I am having issue with using pyspark in Apache-Zeppelin (version 0.6.0) notebook. Running the following simple code gives me pyspark interpreter not found error

%pyspark
a = 1+3

Running sc.version gave me res2: String = 1.6.0 which is the version of spark installed on my machine. And running z return res0: org.apache.zeppelin.spark.ZeppelinContext = {}

  1. Pyspark works from CLI (using spark 1.6.0 and python 2.6.6)

  2. The default python on the machine 2.6.6, while anaconda-python 3.5 is also installed but not set as default python.

  3. Based on this post I updated the zeppelin-env.sh file located at /usr/hdp/current/zeppelin-server/lib/conf and added Anaconda python 3 path 

export PYSPARK_PYTHON=/opt/anaconda3/bin/python
export PYTHONPATH=/opt/anaconda3/bin/python

After that I have stopped and restarted zeppelin many times using 

/usr/hdp/current/zeppelin-server/lib/bin/zeppelin-daemon.sh

But I can't get the pyspark interpreter to work in zeppelin.

1
pythonapache-sparkpysparkhortonworks-data-platformapache-zeppelin
try %spark.pyspark - Rockie Yang
Now I get error message spark.pyspark interpreter not found - Khurram Majeed
I just downloaded 0.6 now and it works for me. sc.version is 1.6.1. there should have some indication in zeppelin/logs/zeppelin-interpreter-spark-*.log - Rockie Yang
The exports did the trick for me with this setup, github.com/dylanmei/docker-zeppelin/blob/master/examples/… - ElMesa

1 Answers

1
votes

To people who found out pyspark not responding, please try to restart your spark interpreter in Zeppelin,it may solve pyspark not responding error.