AWS EMR perform "bootstrap" script on all the already running machines in cluster

17
votes

I have one EMR cluster which is running 24/7. I can't turn it off and launch the new one.

What I would like to do is to perform something like bootstrap action on the already running cluster, preferably using Python and boto or AWS CLI.

I can imagine doing this in 2 steps:

1) run the script on all the running instances (It would be nice if that would be somehow possible for example from boto)

2) adding the script to bootstrap actions for case that I'd like to resize the cluster.

So my question is: Is something like this possible using boto or at least AWS CLI? I am going through the documentation and source code on github, but I am not able to figure out how to add new "bootstrap" actions when the cluster is already running.

1
pythonamazon-web-servicesbotoemramazon-emr
Would someone who has rep on emr or amazon-emr please propose that these tags be synonymous? I do not have any rep on these tags to propose a synonym, but they're clearly the same. – C8H10N4O2

1 Answers

6
votes

Late answer, but I'll give it a shot:

That is going to be tough.

You could install Amazon SSM Agent and use the remote command interface to launch a command on all instances. However, you will have to assign the appropriate SSM roles to the instances, which will require rebuilding the cluster AFAIK. However, any future commands will not require rebuilding.

You would then be able to use the CLI to run commands on all nodes (probably boto as well, haven't checked that).