OpenCover takes much longer to run than the nunit-console

Question

I'm trying to add unit tests to this project: https://github.com/JimBobSquarePants/ImageProcessor

When running the unit tests, they take maybe 1 or 2 minutes to run (it's an image processing library, and I don't expect them to be insanely fast).

The problem is that when I run OpenCover over these tests, they take something like 20 minutes to run.

The gist of the current unit tests is that there are a bunch of test images, and each unit test (more like integration tests, actually) reads each image, and runs a bunch of effects on it.

I'm guessing that I'm doing something wrong, but what? Why does it takes so much more time on OpenCover than NUnit runner ?

Answer 1

OpenCover instruments the IL of your assemblies (for which it can find a PDB file - because that is where the file location information is kept) and then for each sequence point (think of places you can put a break point) and each conditional branch path will cause an action to register the visit (and increase the visit count).

For algorithmic code you will find running coverage on heavy integration tests will be a performance issue so make sure you only run coverage on tight integration tests or on unit tests e.g. in your case perhaps use small images (as previously suggested) that can test the correctness of your code.

You haven't described how you are running OpenCover (or which version - I'll assume latest) but make sure you have excluded the test assemblies and are only instrumenting the target assemblies.

Finally OpenCover uses a few queues and threads but if you throw a lot of data at it due to loops etc then it will need time to process the data so it works much better on machines with 4 or more cores. When you are running your tests have a look at the task manager and see what is happening.

Answer 2

This is speculation because I don't use OpenCover, but a coverage analysis tool is supposed to instrument all lines it passes through. Since you are doing image manipulation, each pixel will certainly trigger OpenCover to do some analysis on the matching code lines, and you have lots of pixels

Let's say OpenCover takes 0.01ms to instrument one line of code (again this is pure speculation), that you are working with 1280*1024 images and that each pixel needs 3 lines of code (cap red channel, xor green and blue, whatever), you get 1310720 * 0.01 * 3 = approximately 39 seconds. For one test.

I doubt you only have one test, so multiply this by the amount of tests; you may have an idea of why it is slow.

You should perhaps try testing your algorithms on a smaller scale: unless you are doing image wide operations (I don't see which ones?) you code don't need the whole image to work on. Alternatively use smaller images?

---

EDIT: I had a look at the test suite here and (one again, not knowing OpenCover itself) can say that the problem comes from all the data you are testing; evey single image is loaded and processed for the same tests, which is not how you want to be unit testing something.

Test loading each image type into the Image class for the lib, then test one rotation from an Image class, one resize operation, etc. Don't test everything everytime!

---

Since the tests are necessary, maybe you could explore the OpenCover options to exclude some data. Perhaps refining your coverage analysis by instrumenting only the outer shell of your algorithm would help. Have a look at filters to see what you could hide in order to make it run acceptably.

Alternatively you could run the code coverage only daily, preferently at night?