VSTS Build agent issues in step 'Publish Artifact: drop'

Question

We are seeing inconsistent failures in our vsts builds. These showed up out of nowhere.

• The agent runs on-prem. When using the hosted build agent we don't see any issues.
• On consecutive builds, a lot of retries are needed. Usually the build fails, but sometimes it doesn't.
• Multiple build definitions in different projects are affected

We are trying to find the root cause, any help diagnosing this issue would be much appreciated!

Here's the relevant version information:

[2017-04-25 07:25:58Z INFO AgentProcess] Agent is built for Windows - win7-x64. [2017-04-25 07:25:58Z INFO AgentProcess] RuntimeInformation: Microsoft Windows 6.3.9600 . [2017-04-25 07:25:58Z INFO AgentProcess] Version: 2.111.1

Here is an excerpt from the logs:

017-04-25T07:28:14.1561247Z ##[section]Starting: Publish Artifact: drop 2017-04-25T07:28:14.1561247Z ============================================================================== 2017-04-25T07:28:14.1561247Z Task : Publish Build Artifacts 2017-04-25T07:28:14.1561247Z Description : Publish Build artifacts to the server or a file share 2017-04-25T07:28:14.1561247Z Version : 1.0.40 2017-04-25T07:28:14.1561247Z Author : Microsoft Corporation 2017-04-25T07:28:14.1561247Z Help : More Information 2017-04-25T07:28:14.1561247Z ============================================================================== 2017-04-25T07:28:15.0779877Z ##[section]Async Command Start: Upload Artifact 2017-04-25T07:28:15.0779877Z Uploading 8 files 2017-04-25T07:28:20.1111153Z Total file: 8 ---- Processed file: 7 (87%) 2017-04-25T07:28:30.2611071Z Total file: 8 ---- Processed file: 7 (87%) 2017-04-25T07:28:35.3611234Z Fail to upload 'E:\Build\agents\scully_work\4\a\Project.Name.Web.zip' due to 'An error occurred while sending the request.'. 2017-04-25T07:28:35.3611234Z System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: The connection with the server was terminated abnormally at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.Http.WinHttpHandler.d__101.MoveNext() --- End of inner exception stack trace --- at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.d__3.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.Http.HttpClient.d__58.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult() at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__45.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult() at Microsoft.VisualStudio.Services.FileContainer.Client.FileContainerHttpClient.d__17.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at Microsoft.VisualStudio.Services.FileContainer.Client.FileContainerHttpClient.<UploadFileAsync>d__16.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.VisualStudio.Services.Agent.Worker.Build.FileContainerServer.<UploadAsync>d__14.MoveNext() 2017-04-25T07:28:35.3611234Z Detail upload trace for file that fail to upload: drop/Project.Name.Web.zip 2017-04-25T07:28:35.3611234Z Begin chunking upload file 'drop/Project.Name.Web.zip', chunk size '4194304 Bytes', total chunks '3'. 2017-04-25T07:28:35.3611234Z Attempt '1' for uploading chunk '1' of file 'drop/Project.Name.Web.zip'. 2017-04-25T07:28:35.3611234Z Generate new HttpRequest for uploading file 'drop/Project.Name.Web.zip', chunk '1' of '3'. 2017-04-25T07:28:35.3611234Z Start uploading file 'drop/Project.Name.Web.zip' to server, chunk '1'. 2017-04-25T07:28:35.3611234Z Chunk '1' attempt '1' of file 'drop/Project.Name.Web.zip' fail to send request to server. Error: System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: The connection with the server was terminated abnormally at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.Http.WinHttpHandler.<StartRequest>d__101.MoveNext() --- End of inner exception stack trace --- at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.<SendAsync>d__3.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult() at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.d__45.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult() at Microsoft.VisualStudio.Services.FileContainer.Client.FileContainerHttpClient.d__17.MoveNext(). 2017-04-25T07:28:35.3611234Z Backoff 5.16 seconds before attempt '2' chunk '1'

Update: Updated agent to latest stable, and issue still exists:

[2017-04-25 08:33:04Z INFO AgentProcess] Agent is built for Windows - win7-x64. [2017-04-25 08:33:04Z INFO AgentProcess] RuntimeInformation: Microsoft Windows 6.3.9600 . [2017-04-25 08:33:04Z INFO AgentProcess] Version: 2.115.0

Answer 1

That typically means that there's networking issues somewhere between your agent and the service where you're getting interruptions.

Starting with the 2.x agent we aggressively try to handle network conditions. We break each file into chunks (4MB), open an http stream and pipe that file chunk to the service. Each file chunk is given a long time (10 min)

We have retries per chunk and backoff retrying.

Backoff 5.16 seconds before attempt '2' chunk '1'

Then if we're still unable to upload a chunk even with retries, we put that file into a retry queue for one more pass.

So as you can see, we try really hard.

The more networking interruptions and the larger a file increase the odds. Also if upload speeds are severely slow it could fail to upload chunks in time (4mb in 10 min is a long time) but in your case, it looks like the stream / network is getting interrupted.

You should check the networking. Something is getting interrupted between your machine and the service.

One more experiment to try is a private agent on an private cloud VM (like azure). That gives you a faster cloud network on upload but also the control of a private agent (install the software you want, incremental source and packaging etc...).

Are you seeing the retries I am describing? I'm also forwarding this to our agent dev. I'll edit if there's anything else to add.