How to break an Azure DevOps build if code quality drops

2
votes

I have an Azure DevOps build pipeline implemented in YAML, and the pipeline contains unit tests. If a unit test fails, the build fails - so far so good.

Given that the pipeline is in the same repo as the code, developers are free to modify it as they wish.

Sometimes developers are lazy or in hurry and prefer to get rid of the failing unit tests instead of fixing them. How can I ensure that developers do not remove the unit test build task from the YAML pipeline?

In TeamCity I would add a build failure condition that would compare the number of unit tests in the current build with the previous build, and if it (significantly) drops, it would fail the build. Is there a similar option in Azure DevOps?

1
azure-devopsazure-pipelines
You might be able to do this by using the Gate option (only available in Release defintions, not in Build definitions!) and add a gate "Invoke REST API" as a post-deployment condition. Via Rest API call you could query the underlying database (I've got no experience with that, but I think it might be possible). Your best bet is probably to implement a PS script that counts the entries of the produced test results (via trx file(s)) and compares them to the previous build and then fails either directly or writes the deviation to a custom build variable and your build fails per custom condition. – DanDan
See here link for the custom build conditions and here link and here link for how to write custom build variables using PowerShell. There's a special command like Write-Host ##so[task.setvariable... that let's you do that. By using that you might be able to directly set the build status to failed. – DanDan
I agree with @DanDan: check that the step was available and enabled in a gate. You can search for the setup via the REST API's – Rob Bos

1 Answers

1
votes

After investigating the problem it turned out that there is no easy technical solution for this currently in Azure DevOps.

Yes, for Release Pipelines you can use the Gates, but they are not available for Build Pipelines. Yes, the Timeline REST API can be useful to query whether a particular build contained a specific task, but the question still remains: when to call the REST API to break the build pipeline?

One solution is to trigger two separate build pipelines: one that contains the compilation steps, and implemented in YAML so developers can change it, and one that contains the test steps and implemented in the Visual Designer, so developers can not modify it.

Thanks @DanDan and @Rob Bos for your help, but honestly I feel all these solutions too complicated. And the more I think about the original problem the more I believe this issue should not be solved by the technology. This is a cultural problem: if the devs in the team do not believe in the importance of the tests, they sooner or later will find a way to bypass them. On the other hand if they see the benefit of the tests, they will not ignore them even if that would only required unchecking a single checkbox.

If the team is not on that maturity level that we can trust them, then we should avoid YAML and stick to the Visual Designer.