EMR : All slaves in the job flow were terminated due to Spot

1
votes

We are having one issue with EMR and Spot instances.

We have clusters in different environment (different AWS accounts) in same region :

  • One master node with market type : On Demand
  • Two core nodes with market type : Spot

When the spot instances are terminated (over my maximum bids, out of capacity or wathever), the cluster terminate and I've only this message : All slaves in the job flow were terminated due to Spot

After research, people already had this issues but it was due to a master node with market type Spot, this is not my case :

I tryied to find a response in AWS documentation but its all tell the opposite of what we suspect :the two core nodes termination, terminates the cluster

Regards,

1
amazon-emr

1 Answers

4
votes

This has happened because you have chosen core node to be of spot type. If you read the best practices for instance type in AWS EMR, you will find that they suggest using at least one on-demand instance for the core node. Remember that this will come at an extra cost.

You can use instance fleet option for the core node and add both the spot and on-demand instance type to this instance fleet.

So the general thumb rule is

Keep master and core instances as on-demand and task instances as spot.

I am adding a few links where you can read more about this and configure your cluster accordingly.

Link1: Cluster configuration and Best Practices

Link2: Types of nodes in EMR