Deleting Versions in S3 Bucket AWS

1
votes

I am unable to delete a bucket with 160million+ versions that were created by a bug that was updating a profile picture and since we had versioning set up it caused a heap of a mess. I tried to use the AWS website to delete the versions and delete the s3 bucket but the auth token expires before it runs through all the files and signs me out of AWS. I then tried to use AWS CLI and am confronted with another issue. The command and error can be found below. 

aws s3api delete-bucket --bucket pinch-profile-picture --region us-east-2

and received the following error:

An error occurred (BucketNotEmpty) when calling the DeleteBucket operation: The bucket you tried to delete is not empty. You must delete all versions in the bucket.

AWS does not want to touch or delete the bucket for us because they would be held liable for deleting something.

The following steps I might be able to take are:

1)Create a script to delete everything.

2)Figure out how to delete all the versions from CLI

3)Figure out how to not prevent our token from expiring on the site

4)"Insert your suggestion here"

..

99)manually delete but this is limited to 300 per page which would mean i would be required to delete 300 items 666,666 times if we created 200million versions. So thats a no-go.

But I am open to suggestions, what are your tips to getting the S3 bucket deleted along with the versions. If you've been in the situation before or have experience building scripts please help a brotha out.

Best Regards,

Akshay Kumar

3
amazon-web-servicesamazon-s3aws-cli
The files are 0byte files. – Akumaruwo
Did you try deleting it in the Amazon S3 console? – John Rotenstein

3 Answers

2
votes

If force-deleting the bucket doesn't work, you could Create a Lifecycle Policy for an S3 Bucket.

Configure it to delete all versions of objects and then, within 24 hours, the objects and versions will be gone!

You can then delete the bucket.

0
votes

The remove bucket command with force option on a bucket that has versioning enabled will only add a delete marker to the version history and retains all other versions of that object. Your bucket will not be completely empty as object versions are still there and delete bucket command will fail.

To delete all object versions, you can turn show version option on and delete them individually. Of course, in your case with millions of versions in the bucket deleting individual version is not practical.

The following python code will do the job:

import boto3
session = boto3.Session()
s3 = session.resource(service_name='s3')
bucket = s3.Bucket('your-bucket-name')
bucket.object_versions.delete()
-5
votes

You can try this command:

aws s3 rb s3://mybucket --force

Force parameter: Remove all objects in the bucket and finally it remove itself.

Read More