All tests pass but TFS marks the build as partially succeded

Question

Our current project involves building a robotized box controlled by a .Net application. We interface with quite a few hardware libraries and we did set up a integration server with all the hardware connected to it to run nightly regression tests.

Unfortunately, not all hardware libraires in our setup integrates nicely with TFS and MSTest.

When we run our build and tests with a certain librairy, we have either one of these 2 behaviors:

All tests are martked as passed in TFS but the build is shown as partially succeeded.
TFS shows no test run, but the log shows that all tests passed, build is marked as partially succeeede.

Looking at the logs, we can see these 2 lines after the tests are published to the TFS server, "Handle MSTest Exception" "MSTest.exe returned an exit code of 0 indicating that not all tests passed"

What really puzzles me is that running the same tests with mstest from the command line does not display this problem. Futhermore, when running from the command line, MsTest returns 0 when all tests pass and 1 when there is an error

So my questions are:

What is the appropriate MSTest return code on Success/Failure
Beside the log in visual studio, is there any other detailed loggin feature available?
Any clue on where to look?

Additional info: We did notice a difference when using (or not) the "NoIsolation" flag in mstest. The specific tests will pass when using that flag, however other tests will fail... We are still investigating that one.

Thanks

EDIT: The relevant portion of the log: 30/30 test(s) Passed

Summary
-------
Test Run Completed.
Passed  30
----------
Total   30
Results file:  C:\Builds\1\Galil Daily build\TestResults\D201364-W7$_D201364-W7 2011-07-05 10_23_33_Any CPU_Debug.trx
Test Settings: Default Test Settings
Waiting to publish...
Publishing results of test run D201364-W7$@D201364-W7 2011-07-05 10:23:33_Any CPU_Debug to http://mtlapp07:8080/tfs/DI_DEV...
..Publish completed successfully.
Final Property Values
Category = Galil
CommandLineArguments = /noisolation
Flavor = 
MaxPriority = -1
MinPriority = -1
PathToResultsFilesRoot = C:\Builds\1\Galil Daily build\TestResults
Platform = 
Publish = True
SearchPathRoot = C:\Builds\1\Galil Daily build\Binaries
TestConfigId = -1
TestConfigName = 
TestContainers = System.Linq.OrderedEnumerable`2[System.String,System.String]
TestLists = 
TestMetadata = 
TestNames = 
TestSettings = 
ToolPath = 
Version = -1
Final Property Values
Condition = False
Final Property Values
Condition = True

00:00
Handle MSTest Exception
 MSTest.exe returned an exit code of 0 indicating that not all tests passed.

00:00
If testException is NOT TestFailureException
Initial Property Values
Condition = False
Final Property Values
Condition = False

Edit 2.0:

using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Galil;

namespace TestProject1
{
[TestClass]
public class UnitTest1
  {
    [TestMethod]
    public void TestMethod1()
    {
        Galil.Galil m_galil = new Galil.Galil();
        m_galil.address = "COM4 19200";
        string resp = m_galil.command("MTA=2.5;");
        Assert.AreEqual(":", resp);
        try
        {
            m_galil.address = "OFFLINE";
        }
        catch (Exception)
        {
        }
    }

    [TestMethod]
    public void TestMethod2()
    {
        Galil.Galil m_galil = new Galil.Galil();
        m_galil.address = "COM4 19200";
        string resp = m_galil.command("MTA=2.0;");
        Assert.AreEqual(":", resp);
        try
        {
            m_galil.address = "OFFLINE";
        }
        catch (Exception)
        {
        }
    }
  }
}

Mike Veigel Mike Veigel · Accepted Answer · 2011-07-05T17:36:05

What is the appropriate MSTest return code on Success/Failure

Exit code 1 = not all tests pass

Example:

C:\Program Files\MSBuild\Microsoft\VisualStudio\TeamBuild\Microsoft.TeamFoundation.Build.targets(1377,5,1377,5): warning : MSTest.exe returned an exit code of 1 indicating that not all tests passed.

All tests pass

Example:

Waiting to publish... Publishing results of test run Company_TFSBuild_SVC@BUILD-DEV 2011-07-01 13:15:46_Any CPU_Release to http://company-source:8080/...
..Publish completed successfully.

Beside the log in visual studio, is there any other detailed loggin feature available?

You can get a more detailed log by going to:

Menu Bar -> Tools -> Options... -> Projects and Solutions -> MSBuild project build output verbosity

Any clue on where to look?

Need to see your log file.

All tests pass but TFS marks the build as partially succeded

2 Answers