SparkException: Job aborted due to stage failure

Question

I am fetching the data from HDFS using spark-submit and doing some processing on the fetched data and again trying to write the processed data into HDFS using spark, but getting below exception. I can see that data is getting fetched from HDFS using spark FlatMapFunction and getting the JavaRDDs, but failing with below exception soon after the data is fetched. Below is the command i used to submit the spark job:

spark-submit --class com.xyz.MainClass --master spark://full-hostname-ofspark-master:7077 MyProject.jar

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Master removed our application: FAILED
        at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1033)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1017)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1015)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
        at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1015)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:633)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:633)
        at scala.Option.foreach(Option.scala:236)
        at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:633)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1207)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
        at akka.actor.ActorCell.invoke(ActorCell.scala:456)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
        at akka.dispatch.Mailbox.run(Mailbox.scala:219)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Could you please check no other error or warning appear on the log before this one? A piece of the erroneous code would also be helpful. — Mikel Urkia
i am not finding anything else apart from this exception, but can see few other logs in my application: "2014 45 15 15:45:59 ERROR org.apache.spark.scheduler.TaskSchedulerImpl:Line 74 - Lost executor 3 on xyz.host.com: remote Akka client disassociated 2014 45 15 15:45:59 ERROR org.apache.spark.scheduler.TaskSchedulerImpl:Line 74 - Lost an executor 8 (already removed): remote Akka client disassociated " — MMK
That is what I was looking for, thanks. Could you please tell us which Spark version are you using both on the cluster and on the application (make sure they are identical)? Are you using any native library/function from c++? — Mikel Urkia