AWS S3 cp creates undefined files

Question

While using the aws cli cp command to copy the files recursively, there is a bug which creates some undefined files.

aws s3 cp --recursive $HOME/$MYHOST-$MYTIMESTAMP/$MYHOST-$MYTIMESTAMP-*.xml  s3://mybucket/$MYHOST-$MYTIMESTAMP/

The program works fine and uploads to the specified bucket. But it also creates some undefined files outside the bucket in the root folder. This happens all the time and I have to rm (delete) those annoying undefined files.

I presumed it to be a bug and then tried individually uploading the files rather than using wildcards, with the same results as the recursive, it still creates additional undefined files outside the bucket in the root folder again. And this happens only when I run a bunch of the same cp commands in a bash script. In this case the problem is intermittently showing up.

aws s3 cp  $HOME/$MYHOST-$MYTIMESTAMP/$MYHOST-$MYTIMESTAMP-hello.xml  s3://mybucket/$MYHOST-$MYTIMESTAMP/

However while doing it only for a single file, it doesn't show up. My Cli version -

aws-cli/1.14.34 Python/2.7.14+ Linux/4.4.104-39-default botocore/1.8.38

Any help would be highly appreciated on this.

What does $MYTIMESTAMP resolve to? does it change from command to command? can it be empty, or $MYHOST-$MYTIMESTAMP doesn't resolve to a vaild file name? — Felix
I have custom set those variables to time and host - all that comes out quite fine. The files are also uploading to the relevant folders on s3. thanks for asking — Magnus Melwin
When you say to time, can you elaborate? It looks like the timestamp, at least on the undefined files, changes from run to run. What if you use the actual values? that is, if you run aws s3 cp --recursive /home/magnus/server1-2018-02-12T01:03:16-*.xml s3://mybucket/server1-2018-02-12T01:03:16/ does it work? — Felix
yes its a modified version of the timestamp as shown in the image above that I've used - to keep the files & folder unique. There are restrictions to the naming , you cannot use colon and a few other characters. Please do note that I am uploading files inside uniq folders. not in the root (those undefined files are a byproduct of aws nuisance) — Magnus Melwin

Michael - sqlbot Michael - sqlbot · Accepted Answer · 2018-02-12T12:17:23

You have configured S3 access logging to write logs into this bucket. Presumably, these are the log files for this bucket.

Why the filenames begin with "undefined" is not clear -- something may have gone wrong when you set up logging for the bucket so that the log file prefix did not get saved -- but the filenames look like the names of the log files that S3 creates.

https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html

Best practice is to set up a separate bucket for collecting S3 access logs in each region.

AWS S3 cp creates undefined files

1 Answers