We are seeing Azure Release Pipelines unable to download artifacts on a certain server - it times out after about 2min and we suspect it's a proxy issue i.e. the agent is NOT using the proxy it is configured to use.
When we disable/bypass the artifact download and let the agent run a curl command we see that it also times out when accessing HTTPS URLs but HTTP URLs are fine. So we can reproduce the problem with curl https://curl.haxx.se for example. Running this command directly on the server (as root or any user) works fine so it's an agent issue.
When we tell the agent to export https_proxy=<our proxy> (note httpS) and then CURL, the call to HTTPS succeeds. This implies that CURL knows about our HTTP_PROXY but not about HTTPS_PROXY and curl seems to be ignoring HTTP_PROXY for HTTPS domains. We suspect the agent passes/sets HTTP_PROXY but not HTTP_PROXY!
The problem is we can't run an export command before the artifact download as this is always the first step in the deployment group BEFORE any scripts can be executed.
Aside from the fact that we have export http_proxy and export https_proxy correctly defined in /etc/environment it would seem that the azure pipeline is unaware of the proxy when it tries to download artifact. Our .proxy file contains the proxy as http://proxy.ourlandomain:3333 and it is shown in the agent log as:
Agent is running behind proxy server: 'http://proxy.ourlandomain.com:3333'
...and then later...
[AGENT_PROXYURL] --> [http://proxy.ourlandomain.com:***@ourdomain.com]
It looks somehow garbled. Is it hiding 3333 as a secret with *** and where is this @ourdomain.com coming from???
Finally, the agent is running as root and when we curl HTTPS URLs as root it works fine.
WHY is the agent not using the proxy OR WHY is the artifact download failing?
2020-07-06T13:15:26.6383041Z ##[debug]Evaluating condition for step: 'Download artifact - Test1Artifact - Test1Linux'
2020-07-06T13:15:26.6423849Z ##[debug]Evaluating: succeeded()
2020-07-06T13:15:26.6429537Z ##[debug]Evaluating succeeded:
2020-07-06T13:15:26.6469145Z ##[debug]=> True
2020-07-06T13:15:26.6477846Z ##[debug]Result: True
2020-07-06T13:15:26.6517309Z ##[section]Starting: Download artifact - Test1Artifact - Test1Linux
2020-07-06T13:15:26.6717563Z ==============================================================================
2020-07-06T13:15:26.6718055Z Task : Download pipeline artifact
2020-07-06T13:15:26.6718463Z Description : Download a named artifact from a pipeline to a local path
2020-07-06T13:15:26.6718700Z Version : 1.2.5
2020-07-06T13:15:26.6719046Z Author : Microsoft Corporation
2020-07-06T13:15:26.6719901Z Help : Download a named artifact from a pipeline to a local path
2020-07-06T13:15:26.6720216Z ==============================================================================
2020-07-06T13:15:26.9979976Z Download from the specified build: #81654
2020-07-06T13:15:26.9980651Z Download artifact to: /prg/azure/deploy2/_work/r1/a/Test1Artifact/Test1Linux
2020-07-06T13:15:30.3332655Z ##[debug]Dedup parallelism: 192
2020-07-06T13:17:10.3457179Z ##[error]One or more errors occurred. (A task was canceled.)
2020-07-06T13:17:10.3471565Z ##[debug]Processed: ##vso[task.logissue type=error;]One or more errors occurred. (A task was canceled.)
2020-07-06T13:17:10.3485598Z ##[debug]Processed: ##vso[task.complete result=Failed;]
2020-07-06T13:17:10.3510288Z ##[debug] at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at System.Threading.Tasks.Task`1.get_Result()
at Microsoft.VisualStudio.Services.BlobStore.Common.Telemetry.BlobStoreApplicationInsightsTelemetrySender..ctor(IAppTraceSource tracer, Uri baseAddress)
at Microsoft.VisualStudio.Services.BlobStore.Common.Telemetry.BlobStoreClientTelemetry..ctor(IAppTraceSource tracer, Uri baseAddress)
at Agent.Plugins.PipelineArtifact.DedupManifestArtifactClientFactory.CreateDedupManifestClient(AgentTaskPluginExecutionContext context, VssConnection connection, CancellationToken cancellationToken, BlobStoreClientTelemetry& telemetry)
at Agent.Plugins.PipelineArtifact.PipelineArtifactServer.DownloadAsync(AgentTaskPluginExecutionContext context, PipelineArtifactDownloadParameters downloadParameters, DownloadOptions downloadOptions, CancellationToken cancellationToken)
at Agent.Plugins.PipelineArtifact.DownloadPipelineArtifactTaskV1.ProcessCommandInternalAsync(AgentTaskPluginExecutionContext context, CancellationToken token)
at Agent.PluginHost.Program.Main(String[] args)
2020-07-06T13:17:10.3588657Z ##[section]Finishing: Download artifact - Test1Artifact - Test1Linux
UPDATE: Setting Environment=https_proxy=http://proxy.ourlandomain.com:3333 in the systemd config for the agent solves the problem and the artifact is downloaded. It would seem that either the download artifact task is not respecting http_proxy OR the agent should be and isn't setting https_proxy.
