Update AWS S3 object key (key name)

Question

I need to rename a huge number of files stored in an AWS S3 bucket. Each file/object will be renamed individually (based on the certain criteria - there is no common "prefix/suffix"). I came up with the list of 'current_filename' and 'new_filename' which can be executed the CLI as a .bat file/queue.

Is it possible to update just an object key (key name) in AWS S3 bucket via CLI or programmatically, without have to copy/move file as below:

aws s3 cp s3://mybucket/folder/myfilename.jpg s3://mybucket/folder/my_NEW_filename.jpg

I can keep all other object metadata the same, but file name (key name) has to change.

Answer 1

You can also use the AWS S3 Move command to move files to new names rather than copy and delete files. AWS S3 Move

# AWS Example
aws s3 mv s3://mybucket/test.txt s3://mybucket/test2.txt

# Copy file from local to S3
aws s3 cp robots.txt s3://ascisolutions.com

# Rename/move file
aws s3 mv s3://ascisolutions.com/robots.txt s3://ascisolutions.com/robots.txt.bak

# Move file to a directory in S3
aws s3 mv s3://ascisolutions.com/robots.txt.bak s3://ascisolutions.com/test/robots.txt.bak

# Rename/move file
aws s3 mv s3://ascisolutions.com/test/robots.txt.bak s3://ascisolutions.com/test/robots.txt

# Move file back to root directory of S3 bucket
aws s3 mv s3://ascisolutions.com/test/robots.txt s3://ascisolutions.com/robots.txt

Answer 2

Objects/Files in Amazon S3 are immutable and cannot be appended to or changed. In order to simulate append, you would need to write the entire file again with the additional data.

You need to copy to a different object to change its name. It would be efficient if you move between s3 buckets rather than copying locally and moving back.

We ran into a similar situation, how I'm going to change all the files. You need to find the pattern to group all the S3 keys and perform renaming parallelly. This is the beauty of a distributed system.

We stored with date/hour/15 minutes pattern. We were able to perform renaming in parallel by renaming them simultaneously through cli.

Hope it helps.