spark history server does not show jobs or stages

0
votes

We are trying to use spark history server to further improve our spark jobs. The spark job correctly writes the eventlog into HDFS and the spark history server also can access this eventlog: we do see the job in the spark history server job listing but aside from the environment variables and executors everything is empty... Any ideas on how we can make the spark history server show everything (we really want to see the DAG for instance) ? We are using spark 1.4.1.

Thanks.

1
hadoopapache-spark

1 Answers

0
votes

I had a similar issue. I am browsing the history server with port forwarding with ssh. After granting the read permission to all the files in the log directory, they appear in my history server!

cd {SPARK_EVENT_LOG_DIR}
chmod +r * # grant the read permission to all users for all files