OpenCover results not matching TFS numbers

Question

We have VS 2008 project (asp.net). It is under TFS. We have written our unit tests using mock framework(nunit?). Developers can execute the tests on their machine and view the code coverage.

Now, we have upgraded our solution to VS 2012 professional. Being professional edition, it does not have support to execute code coverage and so I have ventured into trying out OpenCover.

Problem is that on build via TFS, code analysis shows say 24% as code coverage, but when I execute OpenCover locally on developer machine, it shows completely different figure. We need to aim for what TFS is reporting as that is monitored by or organization automated ALM compliance engines and developers need to be aware that TFS code coverage % does not fall below X.

My OpenCover synatx is:

OpenCover.Console.exe -register:user -target:"C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\MSTest.exe" -targetargs:"/noisolation /testcontainer:"C:\code\APRRel\UnitTest\bin\Debug\unittest.dll" /resultsfile:C:\Reports\MSTest\APRRel.trx" -filter:"+[*]*" -mergebyhash -output:C:\Reports\MSTest\projectCoverageReport.xml

Here are my outputs from TFS and OpenCover:

TFS result

OpenCover result

How can i go around making opencover report similar statistics than that of TFS?

By default, Vs2012 includes the tests in the coverage which artificially raises the results. — bryanbcook
@bryanbcook can you elaborate a bit please? Do you mean that VS2012/TFS is counting my unittest classes itself also towards code coverage? If so, that that is what my current opencover seems to be doing too. — Saurabh Kumar

Shaun Wilde Shaun Wilde · Accepted Answer · 2013-09-08T23:02:00

You are trying, as they say, to compare apples with pears. The two tools are different and they have differing ways of instrumenting the code. Also the number shown in ReportGenerator is based on the number of covered lines. OpenCover has a different number based on the number of sequence points (this is not a 1:1 relationship with lines more of a n:m).

If you open the actual opencover output you will see a summary line near the top like this

<Summary numSequencePoints="895" visitedSequencePoints="895" numBranchPoints="537" visitedBranchPoints="455" sequenceCoverage="100" branchCoverage="84.73" maxCyclomaticComplexity="8" minCyclomaticComplexity="0" />

The sequence coverage is the one you want to look at as this metric is probably what VS coverage is referring to as block-based but again both tools may differ due to differing instrumentation practices (apples vs pears).

Next you need to consider your filter, +[*]* will include all assemblies that have a PDB file. Try excluding your test assemblies with the following filter +[*]* -[*.Tests]* - assumes your test assemblies end with .Tests.

However, IMO, if you want your devs to know the coverage on the build system to be the same as the desktop then they really should run the same tools in both locations and in the same configuration; debug and release coverage can differ due to the compiler creating different IL and this affects the instrumentation and hence the numbers but they are usually within 0-2% depending on your codebase.

OpenCover results not matching TFS numbers

1 Answers