Test category filter in a TFS 2015 vNext build with the mspec test adapter not filtering tests

Question

I have code base in which the unit tests are written with Machine Specifications leveraging the nuget based test runner Machine.Specifications.Runner.VisualStudio, v2.10 to execute the tests. It works fine from the context of Visual Studio (2015 & 2017) and filtering by Trait works as expected. However, when using the Test Assemblies build step it does not seem to honor the category filter. Is there something special with how the TFS build agent runs the test adapter compared to visual studio?

Example Test

    [Subject(typeof(RetrieveConfiguration))]
    [Tags(Categories.Manual)]
    public class When_hitting_the_general_services_uri : SpecificationContext
    {
        private static TestResult result;

        Establish context = () =>
        {
            IServiceInfo serviceInfo = Substitute.For<IServiceInfo>();
            serviceInfo.Url = "";
            environment.GetService("Confiugration", "Retrieve").Returns(serviceInfo);
            x509Manager.LoadFromSignature(ValidSignature).Returns(LoadFromMachine(ValidSignature));
        };

        Because of = () => error = Catch.Exception(() => result = sut.Execute(new Uri("https://myproductionuri/retrieve"), environment));

        It should_have_the_succeeded = () => result.Result.ShouldEqual(StepResult.Success);
    }

Build Step Configuration

Build Log

...
2017-08-10T20:49:44.8717852Z ##[debug]Calling Invoke-VSTest for all test assemblies
2017-08-10T20:49:44.9655216Z Working folder: E:\B39\BA78\WS\18
2017-08-10T20:49:44.9655216Z Executing C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe "E:\B39\BA78\WS\18\s\Src\Test\Verifier.Reporting.Azure.Storage.Spec\bin\Release\Verifier.Reporting.Azure.Store.Spec.dll"  /TestCaseFilter:"TestCategory=ContinuousIntegration" /EnableCodeCoverage /logger:trx /TestAdapterPath:"E:\B39\BA78\WS\18\s"
2017-08-10T20:49:45.1999042Z Microsoft (R) Test Execution Command Line Tool Version 14.0.25420.1
2017-08-10T20:49:45.1999042Z Copyright (c) Microsoft Corporation.  All rights reserved.
2017-08-10T20:49:45.5592884Z Starting test execution, please wait...
2017-08-10T20:49:56.8721150Z Information: Machine Specifications Visual Studio Test Adapter - Executing tests in E:\B39\BA78\WS\18\s\Src\Test\Verifier.Reporting.Azure.Storage.Spec\bin\Release\Verifier.Reporting.Azure.Store.Spec.dll
2017-08-10T20:50:01.5285749Z Passed   Verifier.Reporting.Azure.Store.Spec.When_publishing_a_report.should_have_succeeded
...

Update 8/25 - added the requested screen shots and feedback

Test Explorer without filtering

Notice there are 16 total tests, the indicates ones starting with when hitting are integration tests which are not expected to run within the context of the build agent.

Test Explorer with filtering on Category

The total number of tests has decreased from 16 to 14. Since the test did not have the requested tag it was dropped from the test run.

Running vs2015 vstest.console.exe

As for running the test outside of visual studio, it would appear that the test runner is experiencing issues loading the test adapter on my dev machine, whereas the adapter runs fine in Visual Studio and on the build agent.

@Patrick-MSFT I added the part of the build log around the execution of the unit tests. There is no error raised as part of the build, just the execution of tests which are not decorated with the specified TestCategory\Trait applied in the filter. The tests do not pass as they references resources which are simply not available to the build agent host. We worked around the problem by added if debug pragmas around the tests but that is not an ideal solution. — Tedford
Which arguments are you passed when you use local VS to filter the tests? The test Filter Criteria in TFS VStest task works the same way as the console option /TestCaseFilter of ` vstest.console.exe`. You test code above did not include the corresponding category info. For more info about the Test Filter Criteria in VStest task please refer this blog: dotnetcatch.com/2016/03/11/… — PatrickLu-MSFT
@Patrick-MSFT That is the problem I am experiencing. The code above does not include the test category that was specified in the test filter criteria yet it is included in the test of test run on the build agent. It work as expected via the Test Explorer in Visual Studio using the filter trait:ContinuousIntegration. I believe that is equivalent to TestCategory=ContinuousIntegration in the 2015 build agent. If so then there is something else which is causing a difference in behavior. — Tedford
Hi Tedford, usually the test category need to be specified to be filter. As for filter in Test Explorer, there is no such option to filter the test. Only a configure continuous integration which actually is not a filter. If you don't mind, please kindly show how did you using the filter trait: ContinuousIntegration in the test of test run via visual studio. More details please see my update answer. — PatrickLu-MSFT

PatrickLu-MSFT PatrickLu-MSFT · Accepted Answer · 2017-08-11T16:23:14

The vstest task is just using vstest.console.exe to execute tests. The test Filter Criteria in TFS VStest task works the same way as the console option /TestCaseFilter of vstest.console.exe.

"TestCategory=ContinuousIntegration"

By above, I did not see such TestCategory name in your code, if we run the test by using command line(vstest.console.exe), we have to specify the matched name, which means at least we name a TestCategory attribute above your test method code i.e, my test method code :

[TestCategory("nine"), TestMethod()]
    public void TestMethod1()
    {
        Assert.AreEqual(1, 1);
    }

I need run it by using code below in command line:

Vstest.console.exe UnitTestvstsada.dll /TestCaseFilter:TestCategory=nine

It will filter the test successfully and will get the same result in TFS build.

As for filter in Test Explorer, there is no such option to filter the test. Only a configure continuous integration which actually is not a filter. If you don't mind, please kindly show how did you using the filter trait:ContinuousIntegration in the test of test run via visual studio.

Afraid using the filter trait:ContinuousIntegration that is not equivalent to TestCategory=ContinuousIntegration in the TFS2015 build agent.

Test category filter in a TFS 2015 vNext build with the mspec test adapter not filtering tests

1 Answers