Unable to restore Glacier deep archive to different S3 bucket

Question

I'm trying to restore the files for which I had enabled life cycle rule to Glacier deep archive. When I am trying to restore to a different directory with below command on AWS CLI it's throwing an error after downloading few files.

Command used to restore the directory:aws s3 cp s3://xxxxxxx/cf-ant-prod/year=2020/ s3://xxxxxxxx/atest/ --force-glacier-transfer --storage-class STANDARD --recursive --profile mfa

Error: An error occurred (InvalidObjectState) when calling the CopyObject operation: Operation is not valid for the source object's storage clas

Answer 1

As mentioned on your other question, the --force-glacier-transfer parameter does not restore objects stored in Glacier. It is simply a way to avoid warning notices.

To retrieve from Glacier Deep Archive you will need to:

• Use restore-object to change the Storage Class to Standard or Standard-IA -- this will take some time to restore
• Copy the file to your desired location

It is not possible to do an instant restore or a Restore+Copy.

Answer 2

As mentioned by John Rotenstein - it appears a simple restore of an object from Glacier must be done "in place" and once restored it can be manipulated (copied) as needed.

I was attempting to do something similar to the question topic via Lambda and I struggled for a while because I found the documentation to be murky regarding the fact that restoreObject() requests are either an SQL Select object restoration OR a simple single object restore... and most significantly which parameters apply to which operational mode.

My goal was to restore an object out of Glacier and to a new location/file name in the same bucket. The documentation strongly suggests that this is possible because there are parameters within OutputLocation that allow the BucketName and Prefix to be specified... as it seems to be the case these parameters only apply to SQL Select object restoration.

The confusing part for me was related to the parameters for the restoreObject() method there isn't sufficient differentiation to know that you can't for example provide the Description parameter when making a simple restore request using the GlacierJobParameters parameter... What was frustrating for me was that I would get errors such as:

MalformedXML: The XML you provided was not well-formed or did not validate against our published schema

There was no indication as to where the published schema is located and Googling for the published schemas yielded no results that seemed to apply to the S3 API... my hope was that I could get out of the API documentation and directly refer to the "published schema"... (published where/how?)

My suggestion would be that the documentation for the restoreObject() method be improved and/or the restoreObject() method is split into a simpleRestoreObject() and an sqlRestoreObject() object so that the parameter schemas are cleanly distinct.

Answer 3

Restoring objects from S3 Glacier Deep Archive (or Glacier, for that matter) must be done individually, and before copying those objects to some other location.

One way to accomplish this is by first retrieving the list of objects in the desired folder using s3 ls, for example

aws s3 ls s3://xxxxxxx/cf-ant-prod/year=2020/ --recursive

and, using each of those object names, running a restore command individually:

aws s3api restore-object --bucket s3://xxxxxxx --key <keyName> --restore-request Days=7

This will initiate a standard restore request for each object, so expect this to take 12-24 hours. Then, once the restores are complete, you are free to copy those objects using your above syntax.

Another option would be to use a tool such as s3cmd, which supports recursive restores given a bucket and folder. However, you'll still have to wait for the restore requests to complete before running a cp command.