S3 Sync issue running in azure devops pipeline on linux

1
votes

I've also asked directly in the GitHub of the project, but I'm also taking a chance since more people look on here.

Confirm by changing [ ] to [x] below:

Issue is about usage on:

  • [x] Service API : I want to do X using Y service, what should I do?
  • [ ] CLI : passing arguments or cli configurations.
  • [ ] Other/Not sure.

Platform/OS/Hardware/Device What are you running the cli on? I'm running the CLI on ubuntu-latest (currently ubuntu-18.04), I've also tried with ubuntu-20.04. This is used in my CD on azure devops

Describe the question I keep running the following command command as part of my CD pipeline and it keeps failing with 255 error code. When I run the same command locally on my machine with the same user profile, it works. My local machine is a macOS 10.15 though.

aws s3 sync /mybuildartifact/ s3://bucketName/subDir/ --delete --profile myUserProfile --debug

I've made sure to check that the CLI properly installed the user profile and that the command has access to it. I've used the following commands to setup the user profile and check it's existence (I'm using the same method to test locally)

aws configure import --csv file://$(serviceUserCredentials.secureFilePath)
aws configure list-profiles

Logs/output Get full traceback and error logs by adding --debug to the command.

2020-11-24 16:24:11,346 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.metadata: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1a9618a320> 2020-11-24 16:24:11,346 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.copy-props: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1a9618a320> 2020-11-24 16:24:11,346 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.sync: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f1a968225f8> 2020-11-24 16:24:11,346 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.metadata-directive: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1a9618a320> 2020-11-24 16:24:11,346 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.size-only: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1a9618a320> 2020-11-24 16:24:11,346 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.sync: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f1a968225f8> 2020-11-24 16:24:11,346 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.exact-timestamps: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1a9618a320> 2020-11-24 16:24:11,346 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.sync: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f1a968225f8> 2020-11-24 16:24:11,346 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.delete: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1a9618a320> 2020-11-24 16:24:11,346 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.sync: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f1a968225f8> 2020-11-24 16:24:11,346 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/ 2020-11-24 16:24:11,349 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): 169.254.169.254:80 2020-11-24 16:24:11,351 - MainThread - urllib3.connectionpool - DEBUG - http://169.254.169.254:80 "PUT /latest/api/token HTTP/1.1" 400 11 2020-11-24 16:24:11,351 - MainThread - awscli.clidriver - DEBUG - Exception caught in main() Traceback (most recent call last): File "awscli/clidriver.py", line 453, in main File "awscli/customizations/commands.py", line 198, in call File "awscli/customizations/commands.py", line 191, in call File "awscli/customizations/s3/subcommands.py", line 709, in _run_main File "awscli/customizations/s3/subcommands.py", line 480, in _run_main File "awscli/customizations/s3/subcommands.py", line 473, in get_client File "botocore/session.py", line 778, in create_client File "botocore/session.py", line 827, in _resolve_region_name File "botocore/session.py", line 237, in get_config_variable File "botocore/configprovider.py", line 289, in get_config_variable File "botocore/configprovider.py", line 386, in provide File "awscli/utils.py", line 60, in provide File "awscli/utils.py", line 65, in _get_instance_metadata_region File "awscli/utils.py", line 105, in retrieve_region File "awscli/utils.py", line 114, in _get_region File "botocore/utils.py", line 298, in _fetch_metadata_token botocore.utils.BadIMDSRequestError: <botocore.awsrequest.AWSRequest object at 0x7f1a96155160>

1
amazon-s3azure-devopsaws-cli
I keep running the following command command as part of my CD pipeline and it keeps failing with 255 error code. When I run the same command locally on my machine with the same user profile, it works. Azure DevOps use project collection service account to run command in pipeline. Please make sure it has sufficient permission, as same as your locally account. - PatrickLu-MSFT
like I stated, I was using the exact same user profile. I answered my question with the answer I got from contributors to the repository on GitHub. - LostBalloon

1 Answers

2
votes

So I got a response on the GitHub that linked me to another issue about some reason related to how the networking is done. for more details you can see the following issue:

https://github.com/aws/aws-cli/issues/5623

In some cases, like when running the AWS CLI on an Azure instance, another service can be running at the same non-routable IP address used by IMDS. One way to get around this is to explicitly set a region.

As soon as I set a region to the profile I was using to run the command, it worked.

This also explains why I wasn't having issues running locally.