fatal error: An error occurred (404) when calling the HeadObject operation: Key " " does not exist

Question

This is my setup:

I use AWS Batch that is running a custom Docker image
The startup.sh file is an entrypoint script that is reading the nth line of a text file and copying it from s3 into the docker.
For example, if the first line of the .txt file is 'Startup_00001/ Startup_000018 Startup_000019', the bash script reads this line, and uses a for loop to copy them over.
This is part of my bash script:

STARTUP_FILE_S3_URL=s3://cmtestbucke/Config/
Startup_FileNames=$(sed -n ${LINE}p file.txt)
for i in ${Startup_FileNames}
do
        Startup_FileURL=${STARTUP_FILE_S3_URL}$i
        echo $Startup_FileURL
        aws s3 cp ${Startup_FileURL} /home/CM_Projects/ &
done

Here is the log output from aws:

s3://cmtestbucke/Config/Startup_000017
s3://cmtestbucke/Config/Startup_000018
s3://cmtestbucke/Config/Startup_000019
Completed 727 Bytes/727 Bytes (7.1 KiB/s) with 1 file(s) remaining download: s3://cmtestbucke/Config/Startup_000018 to Data/Config/Startup_000018
Completed 731 Bytes/731 Bytes (10.1 KiB/s) with 1 file(s) remaining download: s3://cmtestbucke/Config/Startup_000017 to Data/Config/Startup_000017
fatal error: *An error occurred (404) when calling the HeadObject operation: Key "Config/Startup_000019 " does not exist.*

My s3 bucket certainly contains the object s3://cmtestbucke/Config/Startup_000019
I noticed this happens regardless of filenames. The last iteration always gives this error.
I tested this bash logic locally with the same aws commands. It copies all 3 files.

Can someone please help me figure out what is wrong here?

It looks like there might be a trailing whitespace in the name of the last item. Can you check whether this might be the root cause? — Oleksii Donoha
@OleksiiDonoha there are no whitespaces. However this made me think about the EOL and turns out it was windows(CR LF). I changed it to unix (LF) since the docker image i run is Ubuntu. Now it works. Thank you for the help. — Pooja Hegde

Pooja Hegde Pooja Hegde · Accepted Answer · 2021-06-22T07:35:28

The problem was with EOL of the text file. It was set to Windows(CR LF). The docker image is running Ubuntu which caused the error. I changed the EOL to Unix(LF). The problem was solved.

fatal error: An error occurred (404) when calling the HeadObject operation: Key " " does not exist

1 Answers