Code Coverage Quality Gates for New Lines of Code

0
votes

For Java applications, does anyone have any suggestions on how to build a quality gate that would prevent merging new code that does not have at least 80% code coverage? We're working with a legacy codebase, and while we want to see coverage improve there, we want to stop the bleed and enforce proper test coverage on new code going forward.

I've seen suggestions around using git blame and exclusions/inclusions lists in conjunction with SonarQube or JaCoCo, however this would report on coverage for the entire class, not just the new/changed lines, so we could run into an issue there where merges are blocked due to a lack of legacy coverage.

2
javasonarqubecode-coveragejacoco
"prevent merging new code": my personal opinion is that you should not do this. There are always situations where reaching a coverage goal just does not make sense. To make it obligatory will just waste efforts without any positive effect to quality. - Henry

2 Answers

2
votes

What you're looking for is Coverage on New Code which is included in the default Quality Gate with an 80% requirement.

Unfortunately, since you're looking to head-off merging low-coverage branches into master, that means you're after robust branch analysis. It's in the plans, but we're not there yet. Further, PR analysis (always a first candidate when you're dealing with branches) won't help either because a) it doesn't look at coverage b) it doesn't check the branch's quality gate status (technically there is no QG status for a PR)

As a (somewhat painful altho likely automate-able) workaround:

  1. seed an extra SQ project for the branch with an analysis of master from just before the branch split.
  • Use a sonar.version value of "master"*. This establishes your leak baseline
  • Be sure to use the sonar.branch analysis property to distinguish it from your analysis of master.
  1. configure the branch project's leak period to be since_previous_version
  2. update your branch project analysis job to point to the branch and reset sonar.version to "branch"*
  3. institute a process to check branch QG status before merge
  4. delete the branch SQ project after branch deletion

*Or whatever value makes you happy

For the record, "new code" in this context means both lines added and (old) lines modified since the Leak Period started. Leak Period can be defined as the last X days (sub-optimal), since a given date (better but still not perfect), since a specific version, or since previous_version which will give you a floating leak period that follows version changes. To make sure that new code going into production meets your quality requirements, this last one is recommended but it means that the version string fed into your analysis needs to be kept up to date.

Edit 21June21 Measurement of Coverage on New Code is now available for both branches and PRs in all commercial editions.

1
votes

https://github.com/exussum12/coverageChecker

This tool supports Jacoco format and will only error out for changed lines.

this is written in PHP (so you will need PHP installed on your CI server)

If you generate your Jacoco report as normal and then use 

git diff origin/master... > diff.txt

./diffFilter --jacoco diff.txt jacoco.xml 80

This will fail on builds where the changed lines have less than 80% coverage (and show a list of missed lines), and pass for anything > 80% (also showing a list of missed lines, if applicable)

This tool will also fail for lack of legacy coverage if you modify old code. (You probably should have at least 1 test for the code you are changing). This may slow some changes down but you will gain coverage in all modified areas which will make things better in the future