Here is the case: I launch a spark job, but failed due to many tasks failed for the OOM. So I increase the memory allocation for tasks. I still see some nodes failed for OOM, but the job could succeed at last. My question is how does Spark handle this? It seems that Spark could redistribute the data after a failed try.
P.S. The failed task is apply a Window and Rank operation in the batch job.
Update: I run the batch job in YARN cluster mode. All tasks are configured to have the same memory.
Spark DAGScheduler builds stages of task and submit task to TaskScheduler to run the task. TaskScheduler launch the task on executors and re-run the failed task into different executors and reports to DAGScheduler. You can control the number times, task to be retried before it give up is spark.task.maxFailures. The default value is 4.
Please use repartitions or increase the executor memory in-order to avoid re-run the task.
Spark definitely retry failed tasks on some other executors, but with a limited and configured times at most. If you encounter over 10 or 20 percent OOM of tasks, that really means you allocated too few memory or each partition contains too much data to process. For the later cause, you need to distribute data into smaller partition.
