Running MSTEST.exe /publish on a TeamBuild server, what are the prerequisites?

Question

Similarly to How to use MsTest in Continuous Integration without VS?, I want to run mstest.exe on a TeamBuild server.

My context is Trapping Error Status in MSBuild - i.e., I'm only trying to use mstest.exe /publish to upload the results in to the TFS repository. Thus the full rigmarole in http://www.shunra.com/shunrablog/index.php/2009/04/23/running-mstest-without-visual-studio/ is (you'd hope) likely to be overkill, esp as MSTEST.exe, as covered in http://social.msdn.microsoft.com/Forums/en-US/tfsgeneral/thread/e4575890-8f88-413c-a7f3-0d09d3b9cb01 suggests all I need to do is install Team Explorer.

I've installed VS2008 Team Explorer SP1, and mstest.exe /? still isnt telling me about the publish options.

Recall that it says only Team Explorer is required of mstest.exe /? for publishing:

The following options are also available if Team Explorer is installed:

/publish:[server name] Publish results to the Team Foundation Server.

/publishbuild:[build name] The build identifier to be used to publish test results.

/publishresultsfile:[file name] The name of the test results file to publish. If none is specified, use the file produced by the current test run.

/teamproject:[team project name] The name of the team project to which the build belongs. Specify this when publishing test results.

/platform:[platform] The platform of the build against which to publish test results.

/flavor:[flavor] The flavor of the build against which to publish test results.

I'm going down the procmon.exe path to figure out what's missing, but ideally someone in the know would step in and answer:-

Should mstest.exe /publish only require Team Explorer as stated, or does it require VSTT and Team Explorer?

Answer 1

The /publish option of MSTest.exe does a license check to ensure that a "Team" version of Visual Studio is installed (i.e. the Developer or Test edition) before it does the publish part of the code.

This restriction appears to have been introduced to allow the basic unit testing capabilities in standard versions of Visual Studio but requires you to have one of the more expensive Team editions of Visual Studio to enable the "Team" related features in testing - such as to publish your test results to TFS.

In total there are 3 missing features if you do not have a Team edition of Visual Studio with Team Epxlorer installed.

• Publishing to TFS
• Code coverage
• Connect to remote agent.

If you have the Development Edition installed then you get publish and codecoverage but not the remote agent agent capabilities (i.e. for doing Load lesting).

The Test Edition and Visual Studio Team Suite have everything.

If you would like to see an example of a custom MSBuild task that uses MSTest.exe to publish unit test data from a build server (in this example JUnit test data), then take a look at the Teamprise Build Extensions that I wrote. The source code for these is available under the permissive MS-PL open source license.

Answer 2

I've read on the MSDN forums that the /publish functionality of mstest requires purchasing either VS Team Developer or VS Team Test edition. Any lesser editions such as standard/professional or team explorer don't have it.

I'd really like to see some more evidence though, as it seems fairly unreasonable that with "standard" VS2008 you can write and run unit tests, but you arbitrarily can't run them on your build machine :-(

To answer the question:

Should mstest.exe /publish only require Team Explorer, or does it require VSTT and Team Explorer?

It appears that it requires VSTT (which I assume means visual studio team-tester edition)

Answer 3

Well, the message you get when running mstest.exe /? is actually very misleading. It tells you that Team Explorer is required to publish your test results, but unfortunately installing Team Explorer doesn't change anything. As Martin stated, MsTest does a license check to ensure that appropriate edition of Visual Studio is installed and thus '/publish' and other switches are not available.

And by 'appropriate' edition MsTest means Premium or Ultimate, not sure about other editions.

Microsoft says you should install Visual Studio on your server to have a fully featured CI and this is the only official and supported solution, but if you really don't like this concept (and I don't), you can make MsTest think that Visual Studio is installed on your server.

It is enough to export some registry keys from your development machine to your CI server and voila! - MsTest does publish your test results to TFS. The registry keys are at HKLM\SOFTWARE\Microsoft\VisualStudio\12.0\Licenses or at HKLM\Software\Wow6432Node\Microsoft\VisualStudio\12.0\Licenses depending on your system.

You can find more details of this solution here: http://blog.foxxtrot.net/2010/02/hacking-mstest-out-of-visual-studio.html