We have upgraded our TFS 2013 server to TFS 2015 and we are setting up the new build agents.
Prior to this we performed a trial run and got everything working reasonably well before we did the final conversion of our existing TFS database. The build agent worked perfectly fine.
To our surprise, our build agent no longer co-operates post-upgrade. Creating a trivial build definition and assigning it to the default queue results in an error being raised after 10-15 seconds.
We have tried re-deploying the build agent, play around with permissions and users, but to no avail.
All we get is this in the _diag\ logs:
11:18:09.699993 JobManager.StartJob(job.JobId = a9702f31-2dff-4057-8253-a32ebc106f32)
11:18:09.699993 JobInfo.ctor
11:18:09.699993 JobInfo.ctor - leave
11:18:09.699993 JobManager.StartJob - calling JobWriter.StartJob
11:18:09.699993 JobWriter.StartJob - enter
11:18:09.699993 JobWriter.StartJob - (SKIPPING)first renew
11:18:09.715619 JobWriter.StartJob - start continual renewing
11:18:09.715619 AuthorizationType : OAuth
11:18:09.731245 ---------------------------------------------------------------------------
11:18:09.731245 Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException: Unauthorized
11:18:09.731245 at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.HandleResponse(HttpResponseMessage response)
11:18:09.731245 at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__79.MoveNext()
11:18:09.731245 --- End of stack trace from previous location where exception was thrown ---
11:18:09.731245 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
11:18:09.731245 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
11:18:09.731245 at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__76`1.MoveNext()
11:18:09.731245 --- End of stack trace from previous location where exception was thrown ---
11:18:09.731245 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
11:18:09.731245 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
11:18:09.731245 at Microsoft.VisualStudio.Services.Location.Client.LocationHttpClient.<GetConnectionDataAsync>d__6.MoveNext()
11:18:09.731245 --- End of stack trace from previous location where exception was thrown ---
11:18:09.731245 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
11:18:09.731245 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
11:18:09.731245 at Microsoft.VisualStudio.Services.Client.VssServerDataProvider.<ConnectAsync>d__39.MoveNext()
11:18:09.731245 --- End of stack trace from previous location where exception was thrown ---
11:18:09.731245 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
11:18:09.731245 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
11:18:09.731245 at Microsoft.TeamFoundation.DistributedTask.Agent.Common.ConnectionHelper.GetConnection(Uri serverUri, VssCredentials credentials)
11:18:09.731245 at Microsoft.TeamFoundation.DistributedTask.Agent.JobWriter.StartJob()
11:18:09.731245 at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.HandleResponse(HttpResponseMessage response)
11:18:09.731245 at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__79.MoveNext()
11:18:09.731245 --- End of stack trace from previous location where exception was thrown ---
11:18:09.731245 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
11:18:09.731245 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
11:18:09.731245 at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__76`1.MoveNext()
11:18:09.731245 --- End of stack trace from previous location where exception was thrown ---
11:18:09.731245 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
11:18:09.731245 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
11:18:09.731245 at Microsoft.VisualStudio.Services.Location.Client.LocationHttpClient.<GetConnectionDataAsync>d__6.MoveNext()
11:18:09.731245 --- End of stack trace from previous location where exception was thrown ---
11:18:09.731245 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
11:18:09.731245 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
11:18:09.731245 at Microsoft.VisualStudio.Services.Client.VssServerDataProvider.<ConnectAsync>d__39.MoveNext()
11:18:09.731245 --- End of stack trace from previous location where exception was thrown ---
11:18:09.731245 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
11:18:09.731245 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
11:18:09.731245 at Microsoft.TeamFoundation.DistributedTask.Agent.Common.ConnectionHelper.GetConnection(Uri serverUri, VssCredentials credentials)
11:18:09.731245 at Microsoft.TeamFoundation.DistributedTask.Agent.JobWriter.StartJob()
11:18:09.731245 ---------------------------------------------------------------------------
Running interactively or as a service makes no difference. I have enlisted the build agent's service user as a member of the 'Agent Pool Service Accounts' role.
I believe this is the offending request:
GET https://tfs:8443/tfs/DefaultCollection/_apis/connectionData?connectOptions=IncludeServices&lastChangeId=-1&lastChangeId64=-1 HTTP/1.1
User-Agent: VSServices/14.102.25423.0 (VsoAgent.exe) VsoAgent.exe/1.95.3
Accept-Language: en-US, nb-NO
X-TFS-FedAuthRedirect: Suppress
X-TFS-Session: 7a2e6368-a564-4231-bbd6-xxxxxxxxxx
X-VSS-Agent: VSS: b5d9c453-017f-407c-ac00-b479d0d0e8ed
Authorization: [huge bytesequence]
Host: tfs:8443
Accept-Encoding: gzip
This is rewarded with a 401. My own adminuser however is able to load this URI just fine.
My own admin user is the one I used when I set up the agent. I have also tried launching vsoagent.exe interactively from this user... But no go. Reading between the lines (and looking at some of the roles) there is a user that bares the name of the machine that the agent runs on. I guess this user was created at first and is the one that is actually used. How do I get this situation under control?
EDIT: If I run vsoagent.exe interactively from a user that is NOT included in the pool's list of "Agent Pool Service Accounts", then the agent immediately errs out with Access denied. admrunem needs Listen permissions for pool Regular to perform the action. For more information, contact the Team Foundation Server administrator.. Adding admrunem to the list gets me a little further, namely to the error message that I am trying to diagnose (The "unauthorized" exception). Note: It uses NTLM authorization at this point, then for the fatal call seems to switch to OAUTH.
