Glacier and S3 using lifecycle rules, specified key does not exist

Question

I'm very confused about lifecycle rules with AWS Glacier. I was under the impression I could upload a file to S3, have it go to glacier, and then delete it from S3 and retrieve it from glacier later.

I'm using a clojure wrapper for the AWS SDK and using the method "restore-object" (aws/restore-object credentials bucket-name _key restore-method)

The method call appears to be correct because when I call it on a file which has not been archived to S3, I get an error that the key is not of storage class glacier. But when I call it on a file which has already been archived and deleted, the message is

The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: D101BCAC349AF0DA)

So, exactly the key shouldn't exist in S3, because it's archived. Where is the key? How can I see which keys do exist?

I would very much like to jump start this feedback loop. Using the lifecycle rules, the only action i can see is applying a rule to a folder, if expiration is on, the files go away after a day, I can't browse glacier as there's no GUI for that. Not sure if i should be "expiring" objects at all. I assumed "expire" means deleting it from S3 and making it only available in Glacier. After it is in Glacier, would it shows "storage class = glacier " within the S3 interface ?

You cannot change the S3 object's storage class to Glacier manually, it appears. I can change to reduced redundancy, calling restore-object on that still results in

Restore is not allowed, as object's storage class is not GLACIER (Service: Amazon S3; Status Code: 403; Error Code: InvalidObjectState; Request ID: 85A4913F2CE81872)

So how in the world am I supposed to test this?

I'm going to attempt to test this by removing the "expiration" property and just cross my fingers that this time tomorrow, the storage class changes to "Glacier" in S3 and I can continue testing. If anyone has any pointers to shorten this feedback look I would be tremendously grateful!

Answer 1

I was under the impression I could upload a file to S3, have it go to glacier, and then delete it from S3 and retrieve it from glacier later.

No.

When S3 moves your objects from one of the S3 storage classes (STANDARD, STANDARD_IA, or REDUCED_REDUNDANCY) into the GLACIER storage class, what you have at that point is an S3 object that S3 has stored in Glacier for you, in a Glacier account that S3 owns and controls, which is not visible to you, for which S3 will bill you Glacier storage rates.

It is still an S3 object, but has the GLACIER storage class.

When you need to access one of these objects, you initiate a restore, which leaves the object in Glacier and makes a temporary copy in S3 that persists in RRS and is downloadable until it expires after the number of days you specified when you requested the restore. Then it's removed from RRS and is at this point (still) in Glacier.

If you delete the object from S3, the object is deleted from Glacier.

If you have a lifecycle policy to expire objects, they're purged from wherever S3 has stored them, including Glacier. If you delete them from the S3 console, they are deleted from wherever they are stored, including Glacier.