Pyspark: Error executing Jupyter command while running a file using spark-submit

9
votes

I am able to run pyspark and run a script on Jupyter notebook. But when I try to run the file from terminal using spark-submit, getting this error:

Error executing Jupyter command file path [Errno 2] No such file or directory

Can anyone help me what's wrong with my config?

I'm using Python 2.7 and Spark 1.6

1
pysparkjupyter-notebookspark-submit

1 Answers

26
votes

These problem occurs because you have set jupyter to run pyspark script . Now you should unset PYSPARK_DRIVER_PYTHON variable command for the same is $ unset PYSPARK_DRIVER_PYTHON then try spark-submit again ./bin/spark-submit examples/src/main/python/streaming/network_wordcount.py localhost 9999.