AWS EMR Presto Cluster Terminated abruptly Error: All slaves in the job flow were terminated due to Spot

0
votes

I am having trouble with AWS EMR PrestoDB. I launched a cluster with master nodes as coordinator and core nodes as workers. Core nodes were spot instances. But, master node was on demand. After 5 weeks of cluster launch, i got this error message

Terminated with errorsAll slaves in the job flow were terminated due to Spot

Is it that if all slaves are terminated will make the cluster itself terminate? I see the spot pricing history, and it didn't reached around the max price I set.

What I have already done? I have checked logs that are dumped to s3. I didn't found any information about the cause of termination. It just said

Failed to visit ... <many directories>
2
amazon-emrpresto
its betteer to spin task nodes as spot instances instead of core nodes. Even though u didnt select hdfs/ hbase, looks like EMR is still spinning them (may be to support consistent view? ---and this consistent view uses hdfs --somehow. HDFS needs the nodes to be available (atleast 3). I am guessing if this can be the reason for ur cluster to go down. Also, we noticed that if EMR is up in production for 6-7 days the nodes develop some kind of weird issues (i dont have them on top of my mind); --in my view EMR should be used more as ephemeral --meaning shut down ur presto often - chen

2 Answers

1
votes

I am answering my own question. As per the presto community, there must be at least one master node up and running in the AWS EMR Presto cluster. But since it got terminated, the whole cluster got terminated.

0
votes

To avoid data loss because of spot pricing/interruption the data needs to be backed up by either snapshot, frequent copy to s3 or leaving EBS volume behind. Ref: https://aws.amazon.com/premiumsupport/knowledge-center/spot-instance-terminate/

Your cluster should be still be up but without task nodes. Under Cluster-> Details -> Hardware you can add the task nodes. Adding task nodes

Similar scenario : AWS EMR Error : All slaves in the job flow were terminated

For using Spot you might want to use instance termination notice and also setup the max price : https://aws.amazon.com/blogs/compute/new-amazon-ec2-spot-pricing/