Build agent is unable to download available workspaces

Question

We recently upgraded TFS from TFS 2015 to TFS 2018. We changed build agent in our infrastructure to new agent version 2.122.1

When developers commit their changesets or planed build is executed, sometimes build is processed as expected, but sometimes we receive an strange error upon retrieving sources from TFS repository.

In build log they are logged as:

2018-01-03T15:01:25.6074314Z Querying workspace information.

2018-01-03T15:01:26.5136788Z ##[error]There is an error in XML document (1, 1).

If I open agent detailed log, I got an following information:

[2018-01-03 15:01:25Z INFO ProcessInvoker] Starting process:

[2018-01-03 15:01:25Z INFO ProcessInvoker] File name: 'tf'

[2018-01-03 15:01:25Z INFO ProcessInvoker] Arguments: 'vc workspaces /format:xml /collection:http://servername:8080/tfs/ProjectCollection/ /loginType:OAuth /login:.,******** /noprompt'

[2018-01-03 15:01:25Z INFO ProcessInvoker] Working directory: 'C:\Agent2017\_work\10\s'

[2018-01-03 15:01:25Z INFO ProcessInvoker] Require exit code zero: 'True'

[2018-01-03 15:01:25Z INFO ProcessInvoker] Encoding web name: windows-1252 ; code page: '1252'

[2018-01-03 15:01:25Z INFO ProcessInvoker] Force kill process on cancellation: 'False'

[2018-01-03 15:01:25Z INFO ProcessInvoker] Process started with process id 3524, waiting for process exit.

[2018-01-03 15:01:25Z INFO JobServerQueue] Try to append 1 batches web console lines, success rate: 1/1.

[2018-01-03 15:01:25Z INFO JobServerQueue] Try to upload 1 log files or attachments, success rate: 1/1.

[2018-01-03 15:01:26Z INFO ProcessInvoker] Finished process with exit code 0, and elapsed time 00:00:00.5240505.

[2018-01-03 15:01:26Z ERR StepsRunner] Caught exception from step: System.InvalidOperationException: There is an error in XML document (1, 1). ---> System.Xml.XmlException: Data at the root level is invalid. Line 1, position 1. at System.Xml.XmlTextReaderImpl.Throw(Exception e) at System.Xml.XmlTextReaderImpl.ParseRootLevelWhitespace() at System.Xml.XmlTextReaderImpl.ParseDocumentContent() at System.Xml.XmlTextReaderImpl.Read() at System.Xml.XmlReader.MoveToContent() at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderTFWorkspaces.Read5_Workspaces() --- End of inner exception stack trace --- at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, Object events) at System.Xml.Serialization.XmlSerializer.Deserialize(TextReader textReader) at Microsoft.VisualStudio.Services.Agent.Worker.Build.TFCommandManager.d__31.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.VisualStudio.Services.Agent.Worker.Build.TfsVCSourceProvider.d__3.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.VisualStudio.Services.Agent.Worker.Build.BuildJobExtension.d__17.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.VisualStudio.Services.Agent.Worker.JobExtensionRunner.d__20.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.VisualStudio.Services.Agent.Worker.StepsRunner.d__1.MoveNext() [2018-01-03 15:01:26Z INFO StepsRunner] Step result: Failed

It seems that agent is trying to get list of available workspaces, but it fails. If I try to execute command under appropriate permissions at computer, where agent is running, I get proper list of workspaces.

Answer 1

1. Try to queue build with system.debug = true, and you'll get more information in the log. Check the log to get detailed error message.
2. Remove all the workspaces created by the build process like the following name: ws_1_2 in VS/Team Explorer, and set the clean option in the Get Source task to true, then try to run the build again.

Answer 2

Finally I found an answer to my problem here.

Problem was caused by cached agent information and when I manually executed vc workspaces /format:xml /collection:http://servername:8080/tfs/ProjectCollection/ command at build machine, sometimes I got workspace list, but sometimes I got warning plus workspace list (result is not expected to be same after each call dependent on cache usage). Due to warning like

Local path C:\Agent2017_work\10\s is mapped both in workspace ws_11_09;ProjectCollection Build Service on server http://rwstfs:8080/tfs/ProjectCollection/ and workspace ws_09_10;ProjectCollection Build Service on server http://rwstfs:8080/tfs/ProjectCollection/. Removing workspace ws_09_10;ProjectCollection Build Service on server http://rwstfs:8080/tfs/ProjectCollection/ from the cache. Please remove conflicting mappings.

agent is not able to parse xml document, because it is preceded by non-sgml part.

So I went to

%LocalAppData%\Microsoft\Team Foundation\7.0\Cache

directory, deleted cache content and problem was gone.

EDIT: Alternative way could be delete cached workspaces using command

tf workspaces /remove:*