Analyzing Code Coverage on code using Fakes fail during TFS build

0
votes

I am having some problems after enabling Code Coverage during our TFS builds. The solution under test contain three test projects and out of 209 tests three tests fail (one in one project and two in another), with the following error:

Test method MyNamespace.MyTest threw exception: 
Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.UnitTestIsolationException: Unexpected error returned by SetDetourProvider in profiler library 'C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\IntelliTrace\14.0.0\Microsoft.IntelliTrace.Profiler.14.0.0.dll'.
Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.IntelliTraceInstrumentationProvider.Initialize()
Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.UnitTestIsolationRuntime.InitializeUnitTestIsolationInstrumentationProvider()
Microsoft.QualityTools.Testing.Fakes.Shims.ShimRuntime.CreateContext()
Microsoft.QualityTools.Testing.Fakes.ShimsContext.Create()
MyNamespace.MyTest () in MyFile.cs: line 224

The stack trace indicates that is connected to the use of Shims. All test pass locally even when running Code Coverage and disabling Code Coverage during build makes the tests pass again. The issue seems slightly related to Microsoft Fakes generates UnitTestIsolationException, but Resharper is not installed on the build servers.

What I have tried:

  1. Different build types: Build vNext, XAML Build with customized or Default Template
  2. Explicitly disabled IntelliTrace using /p:EnableIntelliTrace=true and .runsetting
  3. Added Diagnostic="true" to all .fakes files
  4. Increased Logging Verbosity to get more error information
  5. Run the tests locally on the server through command line

My build server setup:

  • Windows Server 2012 R2
  • Team Foundation Server 2015.3
  • Visual Studio Enterprise 2015.3
2
code-coveragetfsbuildmicrosoft-fakes
Try to go to your build agent machine, launch the Visual Studio Developer Command Prompt, and run: vstest.console.exe MyTestAssembly.dll /EnableCodeCoverage to see what result will you get. – Cece Dong - MSFT
@Cece-MSFT: I am getting the same error. I do however get different number of failed tests dependent on order of the three test assemblies in the command. I get 3 failed tests if I run the same order as the server chooses or just one error I choose another order. – Tore Østergaard
As you mentioned you could get number of tests, did it mean you can get Code Coverage now? Did you use default build process template? Could you describe your project, I'd like to have a test on my side. – Cece Dong - MSFT
@Cece-MSFT: I get different numbers of tests failing, depending on the order of test assemblies, but I get Code Coverage results every time. I see now that the title of my question is wrong - I will update this. We are running with a customized process template, but I believe that the test/codecoverage part is standard. The tests are calling ShimsContext.Create() and overriding two functions to return a hardcoded value in some cases. I will try to make a minimal setup that triggers the problem. – Tore Østergaard
I see the same issue when changing to the default build template. – Tore Østergaard

2 Answers

1
votes

I struggled with the same error message, though in my case it was when running the tests themselves. When executed from the commandline, vstest.console.exe worked flawlessly, but when run from a Bamboo build task, it failed with the message above.

The problem was ultimately the user running the task - the build server used the SYSTEM user. When I ran the task as a normal privilege user, it succeeded.