AWS S3 Sync --force-glacier-transfer

7
votes

Few days back i was experimenting with S3 & Glacier and my data was archived so restoring it back i had to use their expedited service (which costs a lot). I want to move all of my content from one bucket to another bucket in same region same account.

When i try to sync the data it gives following error Completed 10.9 MiB/~10.9 MiB (30.0 KiB/s) with ~0 file(s) remaining (calculatingwarning: Skipping file s3://bucket/zzz0dllquplo1515993694.mp4. Object is of storage class GLACIER. Unable to perform copy operations on GLACIER objects. You must restore the object to be able to perform the operation. See aws s3 copy help for additional parameter options to ignore or force these transfers.

I am using the below command and i was wondering what would it cost me in terms of dollars? Because all of my files storage class is changed to "Glacier" from "Standard". So, i am forced to use --force-glacier-transfer flag

aws s3 sync s3://bucketname1 s3://bucketname2 --force-glacier-transfer --storage-class STANDARD
2
amazon-web-servicesamazon-s3aws-sdkaws-cli

2 Answers

8
votes

If you restored them and are before the expiry-date you should be able to sync them without an additional restore. You get the Glacier error for all recursive commands as the API they use doesn't check to see if they are restored. You can read about it in the ticket where they added the --force-glacier-transfer.

https://github.com/aws/aws-cli/issues/1699

When using the --force-glacier-transfer flag it doesn't do another restore, it just ignores the API saying the object is in Glacier and tries anyways. It will fail if the object is not restored (it won't try to restore it).

Note that this is only with the recursive commands (eg. sync and cp/mv with --recursive), if you just copy 1 file it will work without the force flag.

1
votes
BUCKET=my-bucket
DATE=$1
BPATH=/pathInBucket/FolderPartitioDate=$DATE
DAYS=5
 for x in `aws s3 ls s3://$BUCKET$BPATH --recursive | awk '{print $4}'`;
 do
  echo "1:Restore $x"
  aws s3api --profile sriAthena restore-object --bucket $BUCKET --key $x --restore-request Days=$DAYS,GlacierJobParam
eters={"Tier"="Standard"};
  echo "2:Monitor $x"
  aws s3api head-object --bucket $BUCKET --key $x;
done

https://aws.amazon.com/premiumsupport/knowledge-center/restore-s3-object-glacier-storage-class/