TFS2015 new install, 503 Service Unavailable

3
votes

I've installed TFS 2015 all according the 'Single Server' manual (https://msdn.microsoft.com/en-us/Library/vs/alm/TFS/setup/single-server?f=255&MSPPError=-2147217396).

I configured one simple build, building a small console-app with a unit test project added, with one little unit test.

The build configuration has now three steps.

  1. NuGet Installer
  2. Visual Studio Build
  3. Visual Studio Test

Now, every time the third step is called, an error occurs, a red box in the browser with the text:

503: 503: error
TFS.WebApi.Exception: 503: error    at k (http://teamfs2015:8080/tfs/_static/tfs/Dev14.M95.3/_scripts/TFS/min/corejs?v=IiKWXrylMIeJdHBpXJwuE0SpVjVnmci79kLQuyG8-s41:64:375)
at http://teamfs2015:8080/tfs/_static/tfs/Dev14.M95.3/_scripts/TFS/min/corejs?v=IiKWXrylMIeJdHBpXJwuE0SpVjVnmci79kLQuyG8-s41:64:2955
at d (http://teamfs2015:8080/tfs/_static/tfs/Dev14.M95.3/_scripts/TFS/min/corejs?v=IiKWXrylMIeJdHBpXJwuE0SpVjVnmci79kLQuyG8-s41:64:635)
at http://teamfs2015:8080/tfs/_static/tfs/Dev14.M95.3/_scripts/TFS/min/corejs?v=IiKWXrylMIeJdHBpXJwuE0SpVjVnmci79kLQuyG8-s41:64:2888
at l (http://teamfs2015:8080/tfs/_static/tfs/Dev14.M95.3/_scripts/TFS/min/q.js:27:8122)
at http://teamfs2015:8080/tfs/_static/tfs/Dev14.M95.3/_scripts/TFS/min/q.js:27:8348
at t.when (http://teamfs2015:8080/tfs/_static/tfs/Dev14.M95.3/_scripts/TFS/min/q.js:27:3780)
at t.u.promiseDispatch (http://teamfs2015:8080/tfs/_static/tfs/Dev14.M95.3/_scripts/TFS/min/q.js:27:2824)
at http://teamfs2015:8080/tfs/_static/tfs/Dev14.M95.3/_scripts/TFS/min/q.js:27:1649
at MessagePort.t (http://teamfs2015:8080/tfs/_static/tfs/Dev14.M95.3/_scripts/TFS/min/q.js:27:5773)
Session Id: c0c3ef7f-e1db-4bf8-976a-f820a0701334

Yes, the annoying 503: Service unavailable error....

Is there any way to get rid of this error? I'm using domain accounts for the build servers, they are not configured to expire, the accounts are in the administrators group. Etc. etc.

I think I tried everything there is to know about this problem on the internet, but nothing seems to work.

We have the same problem on out TFS2012 installation.

Looking at the event viewer (application), everytime there is the following information just before the problem occurs:

  1. Information: TFS Services

    Application Request Processing Started

    Server Version = Microsoft.TeamFoundation.Framework.Server, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a Service Account = NT AUTHORITY\NETWORK SERVICE Machine Name = TEAMFS2015

    Application Domain: /LM/W3SVC/2/ROOT/tfs-1-131066466082045873

  2. Information: TFS Services

    The application is being shutdown for the following reason: HostingEnvironment Application Domain: /LM/W3SVC/2/ROOT/tfs-1-131066466082045873

  3. Information: TraceLog Profiler

    TraceLog Profiler component shutdown, process w3wp.exe

  4. Information: TraceLog Profiler

    TraceLog Profiler component shutdown, process te.processhost.managed.exe

  5. Error: Perflib

    The Open Procedure for service "ASP.NET_64_2.0.50727" in DLL "C:\Windows\Microsoft.NET\Framework64\v2.0.50727\aspnet_perf.dll" failed. Performance data for this service will not be available. The first four bytes (DWORD) of the Data section contains the error code.

  6. Error: Perflib

    The Open Procedure for service "Lsa" in DLL "C:\Windows\System32\Secur32.dll" failed. Performance data for this service will not be available. The first four bytes (DWORD) of the Data section contains the error code.

  7. Error: PerfNET

    Unable to open the Server service performance object. The first four bytes (DWORD) of the Data section contains the status code.

Update

The 'Microsoft Team Foundation Server Application Pool' is 'stopped' every time. So the problem can be related to the app pool. Or the app pool will stop because of another error....

Update Reading the comment of jessehouwing, there the results from the event viewer: 'Applications and Services Logs' > 'Microsoft-Team Foundation Server' > 'Debug':

  1. Error Total Time:19235.2866 (Delay 0ms) (Duration 19235.2866ms) (Threshold 10ms) (Fast 18) (Truncated 0) VssRequestContext.HostManagement.TaskAgentPoolHub.Connect:19223 VssRequestContext.HostManagement.TaskAgentPoolHub.Connect:-19235 HostManagement.AspNetRequestContext.EndRequest:19235
  2. Error Total Time:19372.701 (Delay 0ms) (Duration 19372.701ms) (Threshold 10ms) (Fast 19) (Truncated 0) HostManagement.AspNetRequestContext.EndRequest:19372
  3. Error Total Time:19235.1277 (Delay 0ms) (Duration 19235.1277ms) (Threshold 10ms) (Fast 19) (Truncated 0) HostManagement.AspNetRequestContext.EndRequest:19235
  4. Error Total Time:19278.2777 (Delay 0ms) (Duration 19278.2777ms) (Threshold 10ms) (Fast 19) (Truncated 0) HostManagement.AspNetRequestContext.EndRequest:19278 5, 6 and 7. information TFS ActivityLog Entry
  5. Error Microsoft.TeamFoundation.Framework.Server.RequestCanceledException: TF400733: The request has been canceled: Client disconnected.. at Microsoft.TeamFoundation.Framework.Server.VssRequestContext.CheckCanceled(Boolean throwIfShutdown) at Microsoft.TeamFoundation.Framework.Server.VssRequestContext.get_ServiceProvider() at Microsoft.TeamFoundation.Framework.Server.VssRequestContextExtensions.GetService[TService](IVssRequestContext context) at Microsoft.TeamFoundation.Framework.Server.ActivityLoggingAttribute.OnActionExecuted(HttpActionExecutedContext actionExecutedContext) at System.Web.Http.Filters.ActionFilterAttribute.OnActionExecutedAsync(HttpActionExecutedContext actionExecutedContext, CancellationToken cancellationToken) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.ActionFilterAttribute.d__5.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.ActionFilterAttribute.d__0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Controllers.ActionFilterResult.d__2.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.AuthorizationFilterAttribute.d__2.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.AuthorizationFilterAttribute.d__2.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Controllers.ExceptionFilterResult.d__0.MoveNext()
  6. Error Microsoft.TeamFoundation.Framework.Server.RequestCanceledException: TF400733: The request has been canceled: Client disconnected.. at Microsoft.TeamFoundation.Framework.Server.VssRequestContext.CheckCanceled(Boolean throwIfShutdown) at Microsoft.TeamFoundation.Framework.Server.VssRequestContext.get_ServiceProvider() at Microsoft.TeamFoundation.Framework.Server.VssRequestContextExtensions.GetService[TService](IVssRequestContext context) at Microsoft.TeamFoundation.Framework.Server.ActivityLoggingAttribute.OnActionExecuted(HttpActionExecutedContext actionExecutedContext) at System.Web.Http.Filters.ActionFilterAttribute.OnActionExecutedAsync(HttpActionExecutedContext actionExecutedContext, CancellationToken cancellationToken) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.ActionFilterAttribute.d__5.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.ActionFilterAttribute.d__0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Controllers.ActionFilterResult.d__2.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.AuthorizationFilterAttribute.d__2.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.AuthorizationFilterAttribute.d__2.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Controllers.ExceptionFilterResult.d__0.MoveNext()

Looking at the 'Details' of the logging, I'm not seeing anything resambling a cause of a problem. Before the errors, there are several log entries like: TaskHubTimelineRecordLog.AppendLogContent, TaskHubTimelineRecordLog.CreateLog, TaskHubTimelineRecordFeeds.PostLines, TaskHubTimelineRecords.UpdateRecords. Then after the 'UpdateRecords', immediately an error from SignalR (which is communication between backend and the frontend page where i start the build.

2
tfstfs-2015
Anything interesting to see in the Application ad Service Logs\Microsoft-Team Foudnation Server\Debug in the Windows Event Viewer of the Application Tier Server? – jessehouwing
The application pool is definitely shutting down, but why? The domain account of the app pool is always the same, and password will not expire. Also tried the 'local network' account, same problem. Trying stuff with timeouts, nothing changes. After 10 seconds of starting the build, it crashes, every time. I know for sure this will be a problem also when our programmers will connect to the new TFS with Visual Studio. The same problem we have with our TFS2012 installation. We hoped the problem would go away when installing an new TFS2015 in Windows2012 R2 – Dennis
Just changed all services etc. to local accounts instead of domain account, but the problem still remains.. – Dennis
The new build system relies heavily on long-poll http requests. Is the TFS server behind an appliance, reverse proxy, VPN or firewall which may be dropping these connections? It looks like there may be a 20 second request timeout configured somewhere... I don't see how these would trigger a pool reset, unless the pool is configured to recycle after a number of errors has been reached. – jessehouwing
The server is part of our local domain, there is no firewall, no reverse proxy, to VPN. Windows firewall is disabled, no virusscanner is running. – Dennis

2 Answers

2
votes

Pffff, finally found the problem.... For the particular solution, we use a UnitTest.runsettings file (see https://msdn.microsoft.com/en-us/library/jj635153.aspx) to we can configure which assemblies should be excluded from the code coverage reports.

I've configured the 'Visual Studio Test' build step to use this file. Execution options: 'Run Settings File'. The runsettings file was the same as the file for this solution in VS2012 (update 4).

For some odd reason TFS2015 (update 2) cannot handle our runsettings file. So by removing the setting mentioned above, the application pools are not stopping anymore....

Just for example, i'm showing the runsettings file here:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
   <!-- Configurations that affect the Test Framework -->
   <RunConfiguration>
      <!-- Path relative to solution directory -->
      <ResultsDirectory>.\TestResults</ResultsDirectory>

      <!-- [x86] | x64
      - You can also change it from menu Test, Test Settings, Default Processor Architecture -->
      <TargetPlatform>x86</TargetPlatform>

      <!-- Framework35 | [Framework40] | Framework45 -->
      <TargetFrameworkVersion>Framework45</TargetFrameworkVersion>

      <!-- [false] | true -->
      <TreatTestAdapterErrorsAsWarnings>false</TreatTestAdapterErrorsAsWarnings>
   </RunConfiguration>

   <!-- Configurations for data collectors -->
   <!-- For code coverage exclusions etc see: http://msdn.microsoft.com/en-us/library/jj159530.aspx.
        Also: https://msdn.microsoft.com/en-us/library/jj159530.aspx -->
   <DataCollectionRunSettings>
      <DataCollectors>
         <DataCollector friendlyName="Code Coverage"
                        uri="datacollector://Microsoft/CodeCoverage/2.0"
                        assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
            <Configuration>
               <CodeCoverage>

                  <!-- Match assembly file paths: -->
                  <ModulePaths>
                     <Exclude>
                        <ModulePath>.*CPPUnitTestFramework.*</ModulePath>
                        <ModulePath>.*fluentassertions.*</ModulePath>
                        <ModulePath>.*utils.test.dll</ModulePath>
                     </Exclude>
                  </ModulePaths>

                  <!-- Match the company name property in the assembly: -->
                  <CompanyNames>
                     <Exclude>
                        <CompanyName>.*microsoft.*</CompanyName>
                     </Exclude>
                  </CompanyNames>
               </CodeCoverage>
            </Configuration>
         </DataCollector>
      </DataCollectors>
   </DataCollectionRunSettings>

   <!-- Adapter Specific sections -->

   <!-- MSTest adapter -->
   <MSTest>
      <MapInconclusiveToFailed>true</MapInconclusiveToFailed>
      <CaptureTraceOutput>false</CaptureTraceOutput>
      <DeleteDeploymentDirectoryAfterTestRunIsComplete>true</DeleteDeploymentDirectoryAfterTestRunIsComplete>
      <DeploymentEnabled>true</DeploymentEnabled>
   </MSTest>
</RunSettings>

To me, nothing is wrong with this file, and it works like it should when using it in VS2015 and VS2013....

Without this file, TFS2015 is reporting the wrong code coverage, because it does not ignore 3rd party nuget packages. So the code coverage with the runsettings file is about 80%, but TFS2015 is now reporting 31% code coverage....

But hey, the issue seems to be resolved... Or should I say, I found a bug in TFS2015 :-)

0
votes

Try the following:

  1. Restart Server
  2. Delete Team Foundation Cache data tfs_data
  3. Change Server Password
  4. Restart all TFS Services present in IIS.