Why doesn't NUnit timeout attribute work?

2
votes

I'm trying to use the NUnit Timeout attribute on a test in Visual Studio, but it doesn't seem to be working. I would expect that this test fails, but instead it runs for 20 seconds and then passes.

Why doesn't it fail?

namespace Tests
{
    using System.Threading;
    using NUnit.Framework;

    [TestFixture]
    public class timeout_tests
    {
        [Test, Timeout(1)]
        public void timeout_test()
        {
            Thread.Sleep(20000);
        }
    }
}

I tried with the NUnit.framework.dll version 3.0.0 and also on the latest version, 3.0.1. The test passes after 20 seconds with both versions.

2
c#nunit
I can't reproduce using exactly your test and the latest NUnit v3 using either the VS Test Adapter or the console runner. Can you run using the console with the --trace=Verbose and post the contents of the trace file that is generated? - Patrick Quirk
Log file: github.com/LVBen/Documents/blob/master/nunit-console_10936.log - LVBen
There should be another log file, likely with your assembly's name in the filename. Can you post that one (put it in your question)? It also looked like you were using the v2.6.2 version of NUnit console and not v3.0. - Patrick Quirk
Oops. The test fails if I use the 3.0 nunit-console. The test does pass when I run it inside of Visual Studio with the Test Explorer even though I am referencing the 3.0.1 nunit.framework.dll. I guess that is because VS is using an older version of the test runner? - LVBen
That's odd. I'm pretty sure you need to use the v3 NUnit Test Adapter to even run the tests, and so I wouldn't expect it to succeed and the console to fail. I believe if you're using the latest version of each the console and VS test adapter you shouldn't see any issues (as I don't). - Patrick Quirk

2 Answers

1
votes

It turns out that with the older NUnit Test Adapter (e.g. version 2.0.0.0), the test passes after 20 seconds, which is not the expected outcome.

After removing the old adapter and installing the NUnit3 Test Adapter, now the test fails immediately, which is the expected outcome.

-1
votes

This appears to be fixed in the latest version of the NUnit adapter, 2.1.1.

I was personally fooled, though, by the fact that I was sending a huge amount of output to the Console (which was captured in the Output window, I believe by the adapter) very quickly. This created a sort of "back log" in the output, and .NET wanted to complete sending the output it had received before the test quit. You can simulate this behavior with the following test:

[Test, Timeout(10000)] // Time out in 10 seconds
public void Test_MyFunction_DoesNotRunForever()
{
    DateTime start = DateTime.Now;
    while (true)
    {
        TimeSpan runTime = DateTime.Now - start;
        System.Diagnostics.Debug.WriteLine("Doing stuff for " + runTime.TotalSeconds + " seconds." + (runTime.TotalSeconds > 10 ? " Buggy! :(" : ""));
    }
}

If you watch it run, you'll notice that it's still printing output like Doing stuff for 2.043 seconds. even after minutes have gone by.

If you don't want to reduce the output, you should slow down the output with some well placed artificial slow downs like Thread.Sleep, preferably without changing your code. Even Thread.Sleep(1) is enough to prevent such a backlog. Some options that come to mind:

  • If you have mocks, you can put a brief pause in one of your mock's call backs
  • Reorganize your code to accept an output sink as an argument, so you can provide a stub that adds a brief pause and then passes the output on.