Unable to launch python scripts on the YARN Hadoop cluster via remote

2
votes

Since a few weeks i tried to submit python scripts via remote access or connecting to the pyspark shell of the YARN cluster.

I am new to the HADOOP world. What I want is submitting spark scripts in my local shell on the external HADOOP cluster.

My situation: External hadoop YARN cluster. Have access to the important ports. I have Windows 7 64 Bit / Python 2.7.9 64 Bit / Spark 1.4.1. The HADOOP cluster is running without any problems.

My problem: Submitting python scripts via remote access on the HADOOP cluster doesnt work.

If i try spark-submit --master yarn-cluster --num-executors 2 --driver-memory 512m --executor-memory 512m --executor-cores 4 ... example.py

It says 

Error: Cluster deploy mode is not applicable to Spark shells.
Exception: Java gateway process exited before sending the driver its port number

So as far as I understand the problem I think the question is

How do I set the yarn-config correctly to connect with my local client (not part of the cluster) to the external YARN cluster.

1
pythonhadoopyarn
I got some new learnings myself. So far i am able to connect to the extern yarn cluster via putty. So connecting on Cluster and activating pyspark works. Submitting pythonscripts is also working. My current problem ist that i have no local "YARN_CONF_DIR"-location. I found the needed files on my cluster and copied them to my local machine (not part of the cluster). I also have access to the importants ports of the cluster but i can't connect to it. My idea is one the one hand that the copied conf files are corrupt or i need to activate something else. - Patrick C.

1 Answers

0
votes

SPARK VERSION 1.6.0 (which is the current version writing this).

Python code cannot be executed in YARN-cluster mode. Python can only executed in a cluster mode on a native spark cluster.

You can switch to use a spark cluster or you re-implement your code in Java or Scala.