How can the gsutil cp command copy files older than x minutes?

Question

The current Powershell script used to archive files and subdirectories recursively to a cloud bucket is as follows:

gsutil rsync -u -r -C M:\backups gs://archive-bucket

https://cloud.google.com/storage/docs/gsutil/commands/cp

This works fine, however I need a filter to only affect files older than x minutes. Based on the documented feature (see link above) of being able to pipe in a file list, I'd like to apply the timestamp rule to generate a filtered list of all files and subdirectories under M:\backups recursively with some script represented by the placeholder "list_generator" into via -I, the standard input, provided by using cp instead of rsync, roughly like this:

list_generator | gsutil -m cp -n -r -c -I gs://archive-bucket

I presume in the above that the argument -r won't be needed as the recursive action will be relocated into the preceding routine represented by list_generator, while the -n and -c arguments need to be retained to prevent destination overwriting and provide error handling.

Please consider accepting an answer or providing feedback, if your question isn't fully answered yet. — mklement0

mklement0 mklement0 · Accepted Answer · 2020-02-27T18:12:29

Your question boils down to how to get a list of the paths of those file in the directory tree of M:\backups whose last-modified timestamp (LastWriteTime) is older than x minutes ago:

$cutoffDate = (Get-Date).AddMinutes(-20) # example: 20 minutes

Get-ChildItem -File -Recurse M:\backups | 
  Where-Object LastWriteTime -lt $cutoffDate |
    ForEach-Object FullName |
      gsutil -m cp -n -c -I gs://archive-bucket

Note:

In Windows PowerShell, you'll have to set preference variable $OutputEncoding to the character encoding that gsutil expects, so that file names with non-ASCII characters are properly passed; in PowerShell [Core] 6+, $OutputEncoding defaults to UTF-8.
In PowerShell [Core] 6+, the ForEach-Object FullName command is no longer required, because the System.IO.FileInfo instances output by Get-ChildItem then consistently implicitly stringify to the value of their .FullName properties.

How can the gsutil cp command copy files older than x minutes?

1 Answers