TeamCity, xUnit.net, MSBuild, and muting tests - unmuted test failures should halt the build?

Question

I'm using TeamCity to build C# projects, and until recently I wasn't able to use TeamCity's "mute failing test" feature, b/c the MSBuild xunit test runner causes failures in the MSBuild script (return code -1) regardless of whether the tests are muted are not.

This change to my MSBuild script allowed muting to work:

<UsingTask
    AssemblyFile="xunit.runner.msbuild.dll"
    TaskName="Xunit.Runner.MSBuild.xunit"/>

<Target Name="Test">
    <xunit Assembly="MyTests\bin\Debug\MyTests.dll" ContinueOnError="true" />
</Target>

The ContinueOnError="true" is the addition (which I've actually parameterized instead of using a hard-coded value).

And now muting DOES work, in that when muted tests fail, my build continues on successfully. And when unmuted tests fail, my build fails.

The issue I'm running into is that when unmuted tests fail, the build continues on, and runs everything - subsequent build steps, etc. - and failure is decided at the end of the build. In my case we're packaging up build artifacts, and automatically deploying to our integration environment - and both of these things should NOT happen when the build fails due to (unmuted) test failures.

I'm at a loss for how to end the build early when unmuted tests fail. Does anyone have any ideas?

crimbo crimbo · Accepted Answer · 2014-02-10T05:03:30

As of 2/2014, this is a known issue in TeamCity. Two related open issues:

http://youtrack.jetbrains.com/issue/TW-23472 - Apparently build failure doesn't check on subsequent steps, it only checks at the end.
http://youtrack.jetbrains.com/issue/TW-16784 - This is the initial problem I tried to fix with ContinueOnError="true"

So the bottom line is, TeamCity test muting is not usable. You can, however use xUnit test skipping eg [Fact(Skip="Explain here why the test is skipped")].

TeamCity, xUnit.net, MSBuild, and muting tests - unmuted test failures should halt the build?

1 Answers