Running Spark on YARN in yarn-cluster mode: Where does the console output go?

12
votes

I followed this page and ran the SparkPi example application on YARN in yarn-cluster mode.

http://spark.apache.org/docs/latest/running-on-yarn.html

I don't see the output of the program at the end (which is the result of the computation in this case). When I run it in the yarn-client mode (--master yarn-client), I see an output like this:

Pi is roughly 3.138796

Where does the standard out go in the yarn-cluster mode?

1
apache-sparkyarn

1 Answers

11
votes

After much poking around, I found this in the spark-0.9.0 doc.

Examine the output (replace $YARN_APP_ID in the following with the "application identifier" output by the previous command) (Note: YARN_APP_LOGS_DIR is usually /tmp/logs or $HADOOP_HOME/logs/userlogs depending on the Hadoop version.)

$ cat $YARN_APP_LOGS_DIR/$YARN_APP_ID/container*_000001/stdout

Pi is roughly 3.13794

I wish they put this instruction in the 1.1.0 documentation too.