AWS Data Pipeline option to auto terminate EMR cluster

Question

I have setup EMR cluster using Data Pipeline but the cluster is getting terminated after my step function status shows completed. I am executing a python script in my step function which spawns a spark job in background.

How I can make my cluster available after running a successful step function on EMR using Data Pipeline service?

Answer 1

use terminateAfter to specify the hours you want to wait after the processing is complete.

Reference:

http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-ec2resource.html

Hope it helps.