Nondeterministic build server behavior with Team Foundation Server 2013

1
votes

As our team has grown the need for a build server grew until it finally became a project we tackled.

We have a single master project which is queued for auto deployment after a commit to the development branch. The problem is that the build server can queue up the same build, same revision and get wildly different results. Sometimes we get a clean build, deploying as expected to our test server. At other times we can queue the a build and get "Unable to copy file C:\Builds\7424\Project Name\Container Build\bin\somerandomreference. Access to the path C:\Builds\7424\Project Name\Container Build\bin\somerandomreference is denied".

Or we get: Could not copy the file "Scripts\somerandom.js" because it was not found.

Or we get "Could not copy ...EntityFrameWork.XML" because it is being used by another process.

These lead me to believe the problem was with our builds overlapping, so we have set the queue to add additional commits until the earlier build completes.

The only antivirus was the default Microsoft Windows Defender on the Windows 8.1 machine the build server is running on. Initially we excluded the build folders, now we have it turned entirely off.

No one uses this machine, it is dedicated to the build server role and runs no other software, only containing installs of the requirements for our build process.

Am I missing any best practices for a build server? Am I over optimistic in my expectation that a build server should build the same branch and version of the source in a reproducible way (even if it was a reproducible failure)?

Update: after deleting the entire build folder and setting back up, we get this (zero errors in the solution compile, but no test results or output):

Exception Message: Error HRESULT E_FAIL has been returned from a call to a COM component. (type COMException)
Exception Stack Trace:    at Microsoft.TeamFoundation.WorkItemTracking.Client.DataStore.DataStoreNative.BeginDataStoreInit(IntPtr handle, String defaultCachePath, String instanceId, Int32 cacheVersion)
   at Microsoft.TeamFoundation.WorkItemTracking.Client.DataStore.Datastore.BeginDataStoreInit(String defaultCachePath, String instanceId, Int32 cacheVersion)
   at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.InitializeInternal()
   at Microsoft.TeamFoundation.Client.TfsTeamProjectCollection.InitializeTeamFoundationObject(String fullName, Object instance)
   at Microsoft.TeamFoundation.Client.TfsConnection.CreateServiceInstance(Assembly assembly, String fullName)
   at Microsoft.TeamFoundation.Client.TfsConnection.GetService(Type serviceType)
   at Microsoft.TeamFoundation.Client.TfsConnection.GetServiceT
   at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContextT
   at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)
   at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)
   at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)
   at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)

Update2: The process seems to be still having issues, but less often. Here is an example of a non-fatal error from this morning:

C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Microsoft.Common.targets (3540): Could not copy "C:\Builds\7419\Policy Tracker\Container Build\src\Stage\packages\Microsoft.Owin.Security.Cookies.3.0.1\lib\net45\Microsoft.Owin.Security.Cookies.dll" to "C:\Builds\7419\Policy Tracker\Container Build\bin\Microsoft.Owin.Security.Cookies.dll". Beginning retry 1 in 1000ms. The process cannot access the file 'C:\Builds\7419\Policy Tracker\Container Build\bin\Microsoft.Owin.Security.Cookies.dll' because it is being used by another process.

I am currently investingating http://blogs.msdn.com/b/visualstudio/archive/2010/12/21/incorrect-solution-build-ordering-when-using-msbuild-exe.aspx as a possible source of the issue (incorrect solution ordering). This applied to a prior version, but the symptoms seem very similar.

1
tfsmsbuild
Did you modify your build process template? It sounds like you've made some modifications that are causing issues with the build process, not a problem inherent in the template. – Daniel Mann
None of the build scripts have been altered. Strangely, that error happened a few times and then has gone away as well (without any intervention on my part, other than time). – Godeke
Do you have pre/post build actions in your project files? – Daniel Mann
No pre/post action. The only adjustments to the build were MSBuild Arguments /p:VisualStudioVersion=12.0 /p:DeployOnBuild=true;PublishProfile=Stage which is a pretty standard auto-deployment for web projects. – Godeke

1 Answers

0
votes

I have resolved this problem via a setting in the build server: MSBuild Multi-Proc on the Process tab. I had this set to True (the default) and was getting the random errors noted in the question. Flipping this to False has allowed several days of checked in code to build, test and deploy exactly as the product was expected to work.

My research is pointing to a a difference between MSBuild and Visual Studio's approach to scanning for dependencies in projects, but I haven't found a way to resolve those differences in my solution file and am attempting to keep this as simple as possible. Switching to single process builds has increased the build time from 3 minutes to 6 minutes, but I find that an acceptable loss when faced with a choice between non-deterministic and deterministic outcomes.