AWS S3 File merge using CLI

Question

I am trying to combine/merge contents from all the files existing in a S3 bucket folder into a new file. The combine/merge should be done by the ascending order of the Last modified of the S3 file.

I am able to do that manually by having hard coded file names like as follows:

(aws s3 cp s3://bucket1/file1 - && aws s3 cp s3://bucket1/file2 - && aws s3 cp s3://bucket1/file3 - ) | aws s3 cp - s3://bucket1/new-file

But, now I want to change the CLI command so that we can do this file merge based on list of as many files as they exist in a folder, sorted by Last Modified. So ideally, the cp command should receive the list of all files that exist in a S3 bucket folder, sorted by Last Modified and then merge them into a new file.

I appreciate everyone's help on this.

Answer 1

Give you some hints.

First list the files in the reverse order of Last Modified.

aws s3api list-objects --bucket bucket1 --query "reverse(sort_by(Contents,&LastModified))"

Then you should be fine to attach the rest commands as you did

aws s3api list-objects --bucket bucket1 --query "reverse(sort_by(Contents,&LastModified))" |jq -r .[].Key |while read file
do
   echo $file
   # do the cat $file >> new-file
done

aws s3 cp new-file s3://bucket1/new-file