Shutdown spark structured streaming gracefully

Question

There is a way to enable graceful shutdown of spark streaming by setting property spark.streaming.stopGracefullyOnShutdown to true and then kill the process with kill -SIGTERM command. However I don't see such option available for structured streaming (SQLContext.scala).

Is the shutdown process different in structured streaming? Or is it simply not implemented yet?

Answer 1

This feature is not implemented yet. But the write ahead logs of spark structured steaming claims to recover state and offsets without any issues.

Answer 2

This Feature is not implemented yet and also it will give you duplicates if you kill the job from resource Manager while the batch is running.

Answer 3

Try this github example code.

https://github.com/kali786516/FraudDetection/blob/master/src/main/scala/com/datamantra/spark/jobs/RealTimeFraudDetection/StructuredStreamingFraudDetection.scala#L76

https://github.com/kali786516/FraudDetection/blob/master/src/main/scala/com/datamantra/spark/GracefulShutdown.scala#L26