AWS CLI - Moving files between different S3 buckets

2
votes

I am trying to move files between two S3 buckets in two different accounts using AWS CLI. I was able to successfully do a test copy of the files. Now my question is, is the scenario below possible through CLI?

Existing structure:

S3 => Videos(bucket) => ID(folder) => list of transcoded video files

S3 => Thumbnails(bucket) => ID(folder) => list of thumbnails

New Structure:

S3 => Company(bucket) => dev/prod(folder) => ID(folder) => Videos(folder) => list of transcoded video files

                                                        => Thumbnails(folder) => list of thumbnails

The ID folder will also have a Thumbnails(folder) => list of thumbnails

So basically combine both S3 buckets data into one bucket in the new structure. Is it possible to do this kind of copying using CLI?? If so, please guide me in that approach.

1
amazon-web-servicesamazon-s3aws-cli
Does the ID(folder) have multiple values, or is it just one folder? That is, are there multiple folders with different IDs, each of which will contain a Videos and Thumbnails folder? - John Rotenstein
There will be multiple folders with different IDs and each will contain Videos and Thumbnails inside it - Vamsi Challa

1 Answers

5
votes

This is not possible in the AWS CLI, since it is not a simple source->destination copy.

You would either need to program your own solution (eg with a Python script) or what I sometimes do is create an Excel spreadsheet with a list of the source files and some formulates to calculate the destination.

The result is that you want to create a formula that says:

aws s3 sync s3://video-bucket/123/ s3://company-bucket/dev/123/videos/
aws s3 sync s3://thumbnails-bucket/123/ s3://company-bucket/dev/123/thumbnails/

The spreadsheet should do that for each ID in the source list.

Try it on a copy of IDs to see if it works. If it does, copy all the commands into a text file and execute it with the shell.