How does Apache Spark handles system failure when deployed in YARN?

8
votes

Preconditions

Let's assume Apache Spark is deployed on a hadoop cluster using YARN. Furthermore a spark execution is running. How does spark handle the situations listed below?

Cases & Questions

  1. One node of the hadoop clusters fails due to a disc error. However replication is high enough and no data was lost.
    • What will happen to tasks that where running at that node?
  2. One node of the hadoop clusters fails due to a disc error. Replication was not high enough and data was lost. Simply spark couldn't find a file anymore which was pre-configured as resource for the work flow.
    • How will it handle this situation?
  3. During execution the primary namenode fails over.
    • Did spark automatically use the fail over namenode?
    • What happens when the secondary namenode fails as well?
  4. For some reasons during a work flow the cluster is totally shut down.
    • Will spark restart with the cluster automatically?
    • Will it resume to the last "save" point during the work flow?

I know, some questions might sound odd. Anyway, I hope you can answer some or all. Thanks in advance. :)

1
hadoopapache-sparkhadoop-yarn
Why don't you try it out? Doesn't look to difficult to do in my opinion. - Thomas Jungblut
@ThomasJungblut if he tries some of this stuff out and spark recovers he won't necessarily know how or why, not to mention trying some of this stuff takes a lot of time, and possibly money, if some one knows the answer right away that can be helpful to people who see this question - aaronman
@ThomasJungblut Spark may have a local mode, but it doesn't emulates yarn. Furthermore I have no hardware yet and want to know as much as possible about spark because I'm not sure whether we will use it or not. The creation of a complete setup (hadoop+spark on single node) is out of scope and costs to much time. Therefore I want to see what the experiences are. - Matthias Kricke
@aaronman I don't use spark, but two questions are answered by their paper. The remaining by the Hadoop design docs. Besides that this is offtopic for stackoverflow I don't think you need a cluster for that. Spin up two vms in the cloud with Whirr and check it out- 30$ max. - Thomas Jungblut
@ThomasJungblut yes the questions are answered by the paper but things may happen differently on yarn, also if you're admitting that it may cost $30 who wants to go and pay that for no reason. This question is totally on topic, it's asking about the recovery mechanisms of spark and if deploying on YARN affects that recovery - aaronman

1 Answers

13
votes

Here are the answers given by the mailing list to the questions (answers where provided by Sandy Ryza of Cloudera):

  1. "Spark will rerun those tasks on a different node."
  2. "After a number of failed task attempts trying to read the block, Spark would pass up whatever error HDFS is returning and fail the job."
  3. "Spark accesses HDFS through the normal HDFS client APIs. Under an HA configuration, these will automatically fail over to the new namenode. If no namenodes are left, the Spark job will fail."
  4. Restart is part of administration and "Spark has support for checkpointing to HDFS, so you would be able to go back to the last time checkpoint was called that HDFS was available."