NUnit 3.2.0 Console Runner System.NullReferenceException when running with /domain=multiple

Question

When running with /domain=multiple in NUnit Console Runner a NullReference exception occurs.

Running without /domain=multiple or with /domain=single works. Running using ReSharper 10.0.2 test runner with "Use Seperate AppDomain" setting checked works the way I want and runs the test assemblies in parallel.

I want to be able to run parameterized tests from multiple assemblies in parallel using the console runner. Since that requires loading static assets in parallel, the tests need to run in multiple AppDomains.

I created a simple unit testing solution to reproduce the issue. There are are two projects. Each has one test class that looks like this:

[TestFixture]
public class UnitTest1
{
    public static IEnumerable Test1Static
    {
        get
        {

            Console.WriteLine($"before sleep 1 - {DateTime.Now}");
            Thread.Sleep(12000);
            Console.WriteLine($"after sleep 1 - {DateTime.Now}");
            return new List<bool> { true, true };
        }
    }

    [Test, TestCaseSource(nameof(Test1Static))]
    public void TestMethod1(bool tc)
    {
        Assert.IsTrue(tc);
    }
}

Here are the console results:

nunit3-console.exe "UnitTestProject1\bin\Debug\UnitTestProject1.dll" "UnitTestProject2\bin\Debug\UnitTestProject2.dll" /domain=multiple

NUnit Console Runner 3.2.0
Copyright (C) 2016 Charlie Poole

Runtime Environment
   OS Version: Microsoft Windows NT 6.1.7601 Service Pack 1
  CLR Version: 2.0.50727.5485

Test Files
    UnitTestProject1\bin\Debug\UnitTestProject1.dll
    UnitTestProject2\bin\Debug\UnitTestProject2.dll


Test Run Summary
  Overall result: System.NullReferenceException: Object reference not set to an instance of an object.
   at NUnit.Common.ColorConsoleWriter.WriteLabel(String label, Object option, ColorStyle valueStyle)
   at NUnit.Common.ColorConsoleWriter.WriteLabelLine(String label, Object option, ColorStyle valueStyle)
   at NUnit.ConsoleRunner.ResultReporter.WriteSummaryReport()
   at NUnit.ConsoleRunner.ConsoleRunner.RunTests(TestPackage package, TestFilter filter)
   at NUnit.ConsoleRunner.Program.Main(String[] args)

Rob Prouse Rob Prouse · Accepted Answer · 2016-03-26T10:57:41

By default, NUnit 3 runs each assembly in a separate process, (the /process=Multiple flag), so the /domain=multiple flag only makes sense when used in combination with /process=InProcess or with /process=Separate because your tests are already in multiple AppDomains, albeit in different processes. If you add either of those flags, it will work as expected.

That said, NUnit should not be crashing in this situation, so please report it on GitHub.

NUnit 3.2.0 Console Runner System.NullReferenceException when running with /domain=multiple

1 Answers