How to terminate a job in standalone and client mode?

1
votes

I use spark-submit to run a job, which has some exceptions, it blocked, so I tried to use ctrl + c to stop the process.

  1. I would like to know if this job is still running on the cluster on not ?

  2. If it is not the right way to kill the job, what's a right way ?

^C18/09/03 19:03:01 INFO SparkContext: Invoking stop() from shutdown hook 18/09/03 19:03:01 INFO SparkUI: Stopped Spark web UI at http://x.x.x.x:4040 18/09/03 19:03:01 INFO DAGScheduler: Job 2 failed: count at xxx.scala:155, took 773.555554 s 18/09/03 19:03:01 INFO DAGScheduler: ShuffleMapStage 2 (count at xxx.scala:155) failed in 773.008 s 18/09/03 19:03:01 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerStageCompleted(org.apache.spark.scheduler.StageInfo@7f6a32f) 18/09/03 19:03:01 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerJobEnd(2,1535994181627,JobFailed(org.apache.spark.SparkException: Job 2 cancelled because SparkContext was shut down)) 18/09/03 19:03:01 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerSQLExecutionEnd(0,1535994181630) 18/09/03 19:03:01 INFO StandaloneSchedulerBackend: Shutting down all executors Exception in thread "main" org.apache.spark.SparkException: Job 2 cancelled because SparkContext was shut down at org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:818) at org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:816) at scala.collection.mutable.HashSet.foreach(HashSet.scala:78) at org.apache.spark.scheduler.DAGScheduler.cleanUpAfterSchedulerStop(DAGScheduler.scala:816) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onStop(DAGScheduler.scala:1685) at org.apache.spark.util.EventLoop.stop(EventLoop.scala:83) at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1604) at org.apache.spark.SparkContext$$anonfun$stop$8.apply$mcV$sp(SparkContext.scala:1781) at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1290) at org.apache.spark.SparkContext.stop(SparkContext.scala:1780) at org.apache.spark.SparkContext$$anonfun$2.apply$mcV$sp(SparkContext.scala:559) at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:215) at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:187) at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:187) at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:187) at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1953) at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:187) at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:187) at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:187) at scala.util.Try$.apply(Try.scala:192) at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:187) at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:177) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:632) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1873) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1886) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1899) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1913) at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:912) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:358) at org.apache.spark.rdd.RDD.collect(RDD.scala:911) at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:290) at org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1.apply(Dataset.scala:2193) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57) at org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2546) at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$execute$1(Dataset.scala:2192) at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collect(Dataset.scala:2199) at org.apache.spark.sql.Dataset$$anonfun$count$1.apply(Dataset.scala:2227) at org.apache.spark.sql.Dataset$$anonfun$count$1.apply(Dataset.scala:2226) at org.apache.spark.sql.Dataset.withCallback(Dataset.scala:2559) at org.apache.spark.sql.Dataset.count(Dataset.scala:2226) at xx.xx.xx.weekLyLoadingIDFA(xx.scala:155) at xx.xx.xx.retrieve(xx.scala:171) at xx.xx.xx.run(xx.scala:65) at xx.xx.xxRunner$.delayedEndpoint$io$xxx$CellRunner$1(xx.scala:12) at xx.xx.xxRunner$delayedInit$body.apply(xx.scala:11) at scala.Function0$class.apply$mcV$sp(Function0.scala:34) at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12) at scala.App$$anonfun$main$1.apply(App.scala:76) at scala.App$$anonfun$main$1.apply(App.scala:76) at scala.collection.immutable.List.foreach(List.scala:381) at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35) at scala.App$class.main(App.scala:76) at xx.xx.xxRunner$.main(xx.scala:11) at xx.xx.xxRunner.main(xx.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 18/09/03 19:03:01 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down 18/09/03 19:03:01 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 18/09/03 19:03:01 INFO MemoryStore: MemoryStore cleared 18/09/03 19:03:01 INFO BlockManager: BlockManager stopped 18/09/03 19:03:01 INFO BlockManagerMaster: BlockManagerMaster stopped 18/09/03 19:03:01 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 18/09/03 19:03:01 ERROR TransportResponseHandler: Still have 1 requests outstanding when connection from xxxxx/xxxx:7077 is closed 18/09/03 19:03:01 INFO SparkContext: Successfully stopped SparkContext 18/09/03 19:03:01 INFO ShutdownHookManager: Shutdown hook called 18/09/03 19:03:01 INFO ShutdownHookManager: Deleting directory /tmp/spark/spark-xxxxxxxxxx

4
apache-spark
Possible duplicate of Spark Kill Running Application – Manoj Kumar Dhakad
what cluster manager are you using with Spark- local/standalone/yarn etc? Also is it client mode or cluster mode? – moriarty007
Answer to part2 (If it is not the right way to kill the job, what's a right way ?) depends on the cluster manager. – moriarty007
@prakharjain standalone and client mode – Leyla Lee
with client mode the ctrl+c should kill the job and not exiting the spark-shell – Tomasz Krol

4 Answers

1
votes

It depends on the resource manager. In my case ctrl+c works fine on yarn, and the job is killed and you still stay in spark-shell. Also you can kill job from the Spark WEB UI or from YARN.

1
votes

If you are running on yarn you can kill the spark app by below command

yarn application -kill applicationId

For spark on stand alone mode use

spark-submit — kill applicationId — master masterurl
1
votes

The logs above shows that SparkContext was shutdown. This means that the Spark job is not running anymore on the cluster.

Since you are running the application in Client mode, So ctrl+c should kill the application in general.

0
votes

When you start the Spark StandAlone Cluster, It's master have a UI on 8080 port. On the Master UI you will see your application in Running Application Tab. Corresponding to every application there is a (KILL) option button associated with that. Just press that button & it will ask you to confirm it. Confirm it to close. In the Image, you can see a running application & there is a kill option associated with it.

Happy Sparking....