Spark Worker node stops automatically

8
votes

I am running Spark Standalone cluster and on submitting the application, the spark driver stops with the following error.

16/01/12 23:26:14 INFO Worker: Asked to kill executor app-20160112232613-0012/0
16/01/12 23:26:14 INFO ExecutorRunner: Runner thread for executor app-20160112232613-0012/0 interrupted
16/01/12 23:26:14 INFO ExecutorRunner: Killing process!
16/01/12 23:26:14 ERROR FileAppender: Error writing stream to file /spark/spark-1.4.1/work/app-20160112232613-0012/0/stderr
java.io.IOException: Stream closed
        at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        at java.io.FilterInputStream.read(FilterInputStream.java:107)
        at org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:70)
        at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39)
        at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
        at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
        at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1772)
        at org.apache.spark.util.logging.FileAppender$$anon$1.run(FileAppender.scala:38)
16/01/12 23:26:14 INFO Worker: Executor app-20160112232613-0012/0 finished with state KILLED exitStatus 143
16/01/12 23:26:14 INFO Worker: Cleaning up local directories for application app-20160112232613-0012

I am newbie to Spark and its processing.Please help me out in this.

2
javaapache-spark
Did you check the access permissions for this /spark/spark-1.4.1/work/app-20160112232613-0012/0/stderr file? Anyway it might be not onlye issue here (why worker kills executor in first place?). - Dawid Pura
Are you working on AWS? - Rami
@DawidPura : Yes it has all rights and im executing as root user. Could it be like its not taking my jar for execution? - Poppy
@Rami No. Im not workng on aws - Poppy
You can check how logs look like from the Master site. - Szymon Roziewski

2 Answers

4
votes

The error is not caused by the java.io.IOException since you can see clearly 16/01/12 23:26:14 INFO Worker: Asked to kill executor app-20160112232613-0012/0. This exception is raised afterwards when spark tries to write the log files, in which you will also observe the cause of your error.

Even if you run with root privilege spark-submit, it is the spark user that writes the files. I am guessing you are running on your laptop this. Try running sudo chmod -R 777 on your spark folder.

2
votes

In my case, the problem was, spark driver was not able to get the dependencies from the executable jar that was submitted. Merged all dependencies and converted them to a single executable. It fixed the issue.

Please bear with my terminologies :)