Test execution error in Visual Studio 2015 (worked in 2013)

Question

We are in the process of upgrading from Visual Studio 2013 Update 5 to Visual Studio 2015 Update 1.

Our Solution has many tests and we use NUnit 2.6.4, along with the NUnit Test Adapter for NUnit 2.x.

When running these tests in Visual Studio 2013, they all run perfectly well.

However, when running in Visual Studio 2015 the first 200 odd tests run, then execution stops. I can then select the tests that have not yet run and successfully execute these. I have the latest ReSharper installed in both VS2013 and VS2015 and it happily executes all the tests.

We've been keeping test-coverage details for each release since the dawn of time, and from Visual Studio's test runner it shows me the number of blocks covered. But ReSharper shows us the number of statements covered. Slightly different values, but they'd mess up our charts.

When test execution fails, it creates a Dump file (it also creates some XML files that just seem to show what DLLs I've installed). I can open this and "debug" it, but it simply shows me a line of code that fails, and the call stack shows only [Managed Code], which means I can't identify the actual test that's causing the issue.

The fact that this works perfectly in VS2013 and in ReSharper running in VS2015 suggests that "it's not our fault", but whilst I'd like to think that, it doesn't help me fix this.

Any ideas?

Thanks

Griff

I think NUnit has moved fully to its 3.0 release, so you should at least upgrade and test again. — Lex Li
2.x claims to be fully compliant with Vs2015. However, we do already have a planned work-item in to upgrade (we have to upgrade the BuildServer at the same time). However, thanks for the suggestion and I'll test this locally in the next day or so. — DrGriff
Are you on VS 2015 RTM or on Update 1? It sounds like a RTM issue, so if you are there, you should ugdate VS. Regarding NUnit, you should wait a bit with that update since the adapter is still ctp, and bugs are being weeded out. — Terje Sandstrøm
You write you're going to Update1, but your issue sounds like a RTM issue. Does this repro if you try another solution? You can download a testprogram here github.com/OsirisTerje/NUnit.VisualStudioAdapter.TestPrograms and use the TestNUnitMultipleTest.sln. It contains >10000 tests. If that works, your setup is correct, then the next thing would be sending that dumpfile to MS. Regarding NUnit, you should wait a bit with that update since the adapter is still ctp, and bugs are being weeded out. — Terje Sandstrøm

DrGriff DrGriff · Accepted Answer · 2016-03-16T16:30:13

Tracked down the problem. We had one class in our Solution that implements IDispose and one of our Unit tests didn't dispose of that class, it just allowed it to go out of scope.

So because the Disposable object hadn't been disposed, the class' Finalizer hadn't been suppressed. The GC therefore called the Finalizer which in turn attempted to access another object that had also gone out of scope, resulting in an exception that crashed the Test Execution Runner.

Interesting that VS2015 running NUnit 2.x crashes, but the identical setup in VS2013 copes fine.

As an aside, when debugging the DUMP file (see above), I realized that the call stack was irrelevant, I just had to put in some defensive coding in the Finalizer.

Test execution error in Visual Studio 2015 (worked in 2013)

1 Answers