How to clean up the S3 files unloaded from RedShift?

Question

I'm using Amazon Redshift to do data analysis, and sometimes I use 'unload' to unload a RedShift table into S3 and then copy the data into another Redshift instance.

Since I only need to specify a prefix of the data file when doing 'unload', and it will create many files in S3 bucket, I need a graceful way to cleanup all those data files. I don't want to enumerate each object in the bucket and check if the key matches the prefix.

Answer 1

It's very easy with s3cmd from s3tools (get your own copy from http://s3tools.org/s3cmd)

First configure this program with a command:

s3cmd --configure

Then the rest is just one command:

s3cmd del s3://your-bucket/some/dir/prefix*