Empty AWS S3 bucket with lifecyle

Question

I'm creating an AWS S3 bucket on which many files will be uploaded.

Since I don't want those file to stay forever, I would like to empty the bucket every month.

I'm using Terraform to do this.

I have the following documentation https://www.terraform.io/docs/providers/aws/r/s3_bucket.html

And the following Terraform configuration:

resource "aws_s3_bucket" "garbage" {
  bucket = "garbage-${terraform.workspace}"
  acl    = "private"

  lifecycle {
    prevent_destroy = false
  }

  lifecycle {
    prevent_destroy = false
  }

  lifecycle_rule {
    id = tmp
    prefix= "tmp/"
    enabled = true

    expiration {
      days = 1
    }
  }

  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm = "AES256"
      }
    }
  }
}

Even with the documentation, I struggle to find how I could indicate that I want the file to be deleted every month in this bucket.

You haven't included an expiration policy. Also, I'm not aware that you can empty the bucket using lifecycle policies. You would typically configure objects to expire 30 days after they were created. You could potentially set a date e.g. 2019-12-31 but you would then have to update that policy at the start of each new month. — jarmod
You can specify how long an object would live in days before being deleted or transitioned (eg to Glacier). It's not an emptying the bucket on a set date type thing but it achieve the same thing - not keeping each object for more than x days. @RobertReynolds have you tried using the lifecycle_rule parameters? What errors did you get if you did? If you didn't get an error what behaviour did you see vs what you expected? — ydaetskcoR
I don't know what to put in the lifecyle_rule to achieve what you said: "on a set date type thing but it achieve the same thing - not keeping each object for more than x days." — Robert Reynolds
@ydaestcoR I've edited my post to add the lifecylce_rule, I'm not sure about what it will do and It's hard to test, as far as I understand it will delete files in the tmp folder rights ? — Robert Reynolds
Given the rule you added, objects with a prefix of tmp/ will expire and be queued for removal some time after they become 1 day old. Note that AWS evaluates lifecycle rules periodically (once per day afaik) and will queue expired objects for removal. Removal will take place some time after that. You are not charged for storage from the time of expiration. — jarmod

jarmod jarmod · Accepted Answer · 2019-12-19T16:48:45

You haven't included an expiration policy.

You would typically configure objects to expire N days after they were created. You can't create a lifecycle policy that implements "empty this bucket at the end of the month". You could potentially set a date e.g. 2019-12-31 when all objects would expire but you would then have to update that policy to reflect the subsequent expiration date.

Here's an example:

lifecycle_rule {
    id = "trash"
    prefix= "trash/"
    enabled = true

    expiration {
        days = 30
    }
}

This says that objects with a prefix of trash/ will expire and be queued for removal some time after they become 30 days old. Note that AWS evaluates lifecycle rules periodically (once per day afaik) and will queue expired objects for removal. Removal will take place some time after that. You are not charged for storage from the time an object expires.

Note that there are other transitions available too, beyond removal, such as moving objects to cheaper storage tiers such as Glacier.

Empty AWS S3 bucket with lifecyle

1 Answers