automating file archival from ec2 to s3 based on last modified date

Question

I want to write an automated job in which the job will go through my files stored on the ec2 storage and check for the last modified date.If the date is more than (x) days the file should automatically get archived to my s3. Also I don't want to convert the file to a zip file for now.

What I don't understand is how to give the path of the ec2 instance storage and the how do i put the condition for the last modified date.

aws s3 sync your-new-dir-name s3://your-s3-bucket-name/folder-name 

Answer 1

Please correct me if I understand this wrong

Your requirement is to archive the older files

So you need a script that checks the modified time and if its not being modified since X days you simply need to make space by archiving it to S3 storage . You don't wish to store the file locally

is it correct ?

Here is some advice 1. Please provide OS information ..this would help us to suggest shell script or power shell script

Here is power shell script

$fileList = Get-Content "c:\pathtofolder"
foreach($file in $fileList) {
    Get-Item $file | select -Property fullName, LastWriteTime | Export-Csv 'C:\fileAndDate.csv' -NoTypeInformation
}

then AWS s3 cp to s3 bucket.

You will do the same with Shell script.

Answer 2

Using aws s3 sync is a great way to backup files to S3. You could use a command like:

aws s3 sync /home/ec2-user/ s3://my-bucket/ec2-backup/

The first parameter (/home/ec2-user/) is where you can specify the source of the files. I recommend only backing-up user-created files, not the whole operating system.

There is no capability for specifying a number of days. I suggest you just copy all files.

You might choose to activate Versioning to keep copies of all versions of files in S3. This way, if a file gets overwritten you can still go back to a prior version. (Storage charges will apply for all versions kept in S3.)