Capturing output/return value from a spark-submit

Question

Is there anyway that i can capture the output of a spark-submit jobs

spark-submit --master yarn --deploy-mode cluster --queue root.dm.xxx --principal [email protected] --keytab ~/$USER.keytab --driver-class-path "/opt/cloudera/parcels/CDH/lib/hive/lib/" --driver-java-options "-Dspark.executor.extraClassPath=/opt/cloudera/parcels/CDH/lib/hive/lib/" --conf "spark.yarn.appMasterEnv.PYSPARK_PYTHON=/dhcommon/dhpython/python/bin/python2.7" --num-executors 12 --executor-memory 15GB --executor-cores 5 --conf spark.yarn.executor.memoryOverhead=3072 --conf "spark.yarn.appMasterEnv.PYSPARK_DRIVER_PYTHON=/dhcommon/dhpython/python/bin/python2.7" --py-files "/home/xxx/eggs/xxx-1.0-py2.7.egg,/home/xxx/eggs/xxx-1.0-py2.7.egg" simple.py

Answer 1

The shell variable $? has the return value of the last command. You could do something like this:

spark-submit --master yarn ....
ret_val=$?  # In case you want to reference the return code more than once
...

This is of course one way of doing it. You can see this blog for other ways.