Amazon S3 object expiration

3
votes

I read a lot about how to expire(delete) amazon S3 object and tried to setup lifecycle rule to do that, however the objects are not removed, wondering what i did wrong.

i have objects on S3 organized like this:

Amazon S3 > my-test-bucket > my-test-org > a.csv, 
                                           b.xml, 
                                           c.xsl...

i need to delete all those files in my-test-bucket if they are 365 days old. there are lots of files there more than 2 years old so with this rule those files should be removed.

in the lifecycle rule, i specified adding filter to limit scope my-test-bucket, and in 'Expiration' section, i selected both 'Current version' and 'Previous versions',

  • expire current version of object after 365 days from object creation
  • permanently delete previous versions after 365 days from becoming a previous version
  • the rule applies to prefix 'my-test-bucket'

my-test-bucket is not versioning-enabled.

After all these are done, i kept waiting and waiting but nothing happened. those old files are still there. my questions are:

  1. when does auto expire happens? every day at midnight or what?
  2. is there a button that i can push to make it expire objects right away?
  3. in my case, i have files under my-test-org instead of right under my-test-bucket, will this rule still work? do i have to apply wildcard in the rule to make sure it check on all files in all org folder?

thanks

1
amazon-s3

1 Answers

5
votes

"the rule applies to prefix 'my-test-bucket'"

This is your error. The prefix refers to the object key prefix. This does not include the bucket name.

The prefix you want is my-test-org/. The trailing slash is important, because without it, your rule would also match other key prefixes, like my-test-organization and my-test-organism which can lead to unexpected matches.

Remember that there are not really folders in S3. The object keys are strings, delimited by / to signify a folder hierarchy, but the folders are not really containers. A file "foo.txt" that appears in folder "bar" actually has an object key of "bar/foo.txt".

The prefix match, in pseudocode, is

if(left(object_key, len(prefix)) == prefix) { # rule matches

Changing the rule to use this prefix, every object with a key beginning with my-test-org/ will be deleted, if over 365 days old. This, of course, includes all subdirectories.

Rule tests appear to start around midnight UTC and the actual purge starts some time after that.