aws command line interface - aws ec2 wait - Max attempts exceeded

8
votes

I am working on shell script, witch does follow:

  • creates snapshot of EBS Volume;
  • creates AMI image based on this snapshot.

1) I use follow command to create snapshot:
SNAPSHOT_ID=$(aws ec2 create-snapshot "${DRYRUN}" --volume-id "${ROOT_VOLUME_ID}" --description "${SNAPSHOT_DESCRIPTION}" --query 'SnapshotId')

2) I use waiter to wait complete state:
aws ec2 wait snapshot-completed --snapshot-ids "${SNAPSHOT_ID}"

When I test it with EBS Volume 8 GB size everything goes well.
When it is 40 GB, I have an exception:
Waiter SnapshotCompleted failed: Max attempts exceeded

Probably, 40 GB takes more time, then 8 GB one, just need to wait.

AWS Docs (http://docs.aws.amazon.com/cli/latest/reference/ec2/wait/snapshot-completed.html) don't have any timeout or attempts quantity option.

May be some of you have faced the same issue?

4
amazon-web-servicesamazon-ec2aws-cli
" It will poll every 15 seconds until a successful state has been reached. This will exit with a return code of 255 after 40 failed checks." - Adam Ocsvari
Thanks for the information, @AdamOcsvari, in that case we will make loop until exit code is 0. - antonbormotov

4 Answers

13
votes

So, finally, I used follow way to solve it:

  1. Create snapshot
  2. Use loop to check exit status of command aws ec2 wait snapshot-completed
  3. If exit status is not 0 then print current state, progress and run waiter again.

# Create snapshot
SNAPSHOT_DESCRIPTION="Snapshot of Primary frontend instance $(date +%Y-%m-%d)"
SNAPSHOT_ID=$(aws ec2 create-snapshot "${DRYRUN}" --volume-id "${ROOT_VOLUME_ID}" --description "${SNAPSHOT_DESCRIPTION}" --query 'SnapshotId')

while [ "${exit_status}" != "0" ]
do
    SNAPSHOT_STATE="$(aws ec2 describe-snapshots --filters Name=snapshot-id,Values=${SNAPSHOT_ID} --query 'Snapshots[0].State')"
    SNAPSHOT_PROGRESS="$(aws ec2 describe-snapshots --filters Name=snapshot-id,Values=${SNAPSHOT_ID} --query 'Snapshots[0].Progress')"
    echo "### Snapshot id ${SNAPSHOT_ID} creation: state is ${SNAPSHOT_STATE}, ${SNAPSHOT_PROGRESS}%..."

    aws ec2 wait snapshot-completed --snapshot-ids "${SNAPSHOT_ID}"
    exit_status="$?"

done

If you have something that can improve it, please share with us.

4
votes

aws ec2 wait snapshot-completed takes a while to time out. This snippet uses aws ec2 describe-snapshots to get the progress. When it's 100% it calls snapshot-completed.

# create snapshot
SNAPSHOTID=$(aws ec2 create-snapshot --volume-id $VOLUMEID --output text --query "SnapshotId")
echo "Waiting for Snapshot ID: $SNAPSHOTID"

SNAPSHOTPROGRESS=$(aws ec2 describe-snapshots --snapshot-ids $SNAPSHOTID --query "Snapshots[*].Progress" --output text)

while [ $SNAPSHOTPROGRESS != "100%" ]
do
  sleep 15
  echo "Snapshot ID: $SNAPSHOTID $SNAPSHOTPROGRESS"
  SNAPSHOTPROGRESS=$(aws ec2 describe-snapshots --snapshot-ids $SNAPSHOTID --query "Snapshots[*].Progress" --output text)
done

aws ec2 wait snapshot-completed --snapshot-ids "$SNAPSHOTID"

This is essentially the same thing as above, but prints out a progress message every 15 seconds. Snapshots that are completed return 100% immediately.

2
votes

you should probably use until in bash, looks a bit cleaner and you don't have to repeat.

echo "waiting for snapshot $snapshot"
until aws ec2 wait snapshot-completed --snapshot-ids $snapshot 2>/dev/null
do
    do printf "\rsnapshot progress: %s" $progress;
    sleep 10
    progress=$(aws ec2 describe-snapshots --snapshot-ids $snapshot --query "Snapshots[*].Progress" --output text)
done
1
votes

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-retries.html

You can set a variable or use the config file to increase the timeouts.

AWS_MAX_ATTEMPTS=100

~/.aws/config
[default]
retry_mode = standard
max_attempts = 6