How to Stop running Spark Streaming application Gracefully?

10
votes

How Do i stop spark streaming? My spark streaming job is running continuously. I want to stop in a graceful manner.

I have seen below option to shutdown streaming application.

sparkConf.set("spark.streaming.stopGracefullyOnShutdown","true")

Spark configuration: available properties

But, how do i update this parameter on a running application?

1
apache-sparkspark-streaming
You cannot set sparkConf of sparkcontext after the creation of sparkContext. - Knight71
What do you mean when you say graceful? Is anything wrong happening when your app stops? - Amit Kumar
I want to stop the application manully. there are two scenarios. i am clear about how to stop it when some error happend, i have it in the code. but if i want to stop manually, i am looking for a mechanism. - AKC
Does this answer your question? How do I stop a spark streaming job? - Tom Zych

1 Answers

16
votes

Have a look at this blogpost. It it the "nicest" way to gracefully terminate a streaming job I have come across.

How to pass Shutdown Signal :

Now we know how to ensure graceful shutdown in spark streaming. But how can we pass the shutdown signal to spark streaming. One naive option is to use CTRL+C command at the screen terminal where we run driver program but obviously its not a good option. One solution , which i am using is , grep the driver process of spark streaming and send a SIGTERM signal . When driver gets this signal, it initiates the graceful shutdown of the application. We can write the command as below in some shell script and run the script to pass shutdown signal :

ps -ef | grep spark | grep | awk '{print $2}' | xargs kill -SIGTERM

e.g. ps -ef | grep spark | grep DataPipelineStreamDriver | awk '{print $2}' | xargs kill -SIGTERM