Failing test running successful in Azure DevOps Pipeline

Question

I have a ASP.NET Web Forms project which is hosted on a Azure DevOps Repository. The Source Control is TFS.

I Created a continuous integration Pipeline in Azure Devops for the Repository. Here is a Image of the Pipeline:

All the Task from the Pipeline are running correctly beside the VSTest-TestAssemblies Task. My NUnit Test is failing in my local Visual Studio, as it should:

But when running my CI, it tells my that no test failed:

Here you can see my settings and the path to my tests of the VSTest-TestAssemblies Task:

This is what the Folder Structure of my Project looks like:

NR.TestAutomation
- NR.TestAutomation
- NR.TestAutomation.Tests
  - bin
  - obj
  - DemoTests.cs
  - NR.TestAutomation.Tests.csproj
- packages
- NR.TestsAutomation.sln

The Output of the CI Run tells me, that the folder exist and it even finds the test:

Source filter: **/*Tests/*.cs
SystemVssConnection exists true

vstest.console.exe "D:\a\9\s\NR.TestAutomation.Tests\DemoTests.cs"

Why are my failing tests not failing inside the Azure DevOps Pipeline?

rickvdbosch rickvdbosch · Accepted Answer · 2020-06-08T09:40:34

You should specify the (output of the) TestProject to run, not the individual files that contain a test. The test runner cannot execute a .cs file.
So in short; the failing test doesn't run successful in an Azure DevOps pipeline... it doesn't run in the pipeline at all.

Test files
Run tests from the specified files. Ordered tests and webtests can be run by specifying the .orderedtest and .webtest files respectively. To run .webtest, Visual Studio 2017 Update 4 or higher is needed. The file paths are relative to the search folder. Supports multiple lines of minimatch patterns.

Source Visual Studio Test task - Arguments

Possibly interesting as well: more info on File matching patterns reference

EDIT:

what would be the proper Source Path to my tests?

Of course that depends on the name of your test project. Looking at your screenshots, you could use **/*.Tests.dll. This recursively matches all the assemblies where the name ends with .Tests.dll.
Since your project's name is NR.TestAutomation.Tests, the assembly should be named NR.TestAutomation.Tests.dll so it will match. It's also ready for future additions of test projects, as long as their name ends with .Tests.

Failing test running successful in Azure DevOps Pipeline

1 Answers