Kill Spark application used by Zeppelin on YARN after X time of inactivity

0
votes

Running Spark on YARN with Apache Zeppelin, this is consumming X % of the queue, even if no-body is really using it.

Since I am running night CRON jobs, I would like kill/quit/exit the Zeppelin Spark application, after X minutes of inactivity, so the queue is free for night jobs.

1
apache-sparkhadoop-yarnapache-zeppelin

1 Answers

1
votes

Killing with cron is a decent working solution for killing jobs at fixed time (say 04 am).

A more elegant solution would be to integrate Zeppelin with Livy (see https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_zeppelin-component-guide/content/zepp-with-spark.html ) as Livy can be configured to terminate (and then release resources) after x minutes of inactivity.