NUnit tests failed on SetUp are not displayed in TeamCity

Question

I use NUnit and TeamCity to run my tests.

Some tests (not all) has actions made in test class constructor. I call these actions "pre-actioins" for validations. So in one test class I have for example 5 validations (tests) and a set of pre-actions.

I noticed that if a suite of tests if failed on the stage of pre-actions executing then TeamCity doesn't display these tests in its report at all (not under any status).

In build log I see error like:

 SetUp Error : {test_name} + error code.

What I expect from TeamCity is to report these tests at least as Ignored.

To compare running tests using TeamCity with running tests using Visual Studio in Visual Studio the result of the same failure condition will be failure for all the test suite. Failure error will be the same for all the tests.

So what I want is just to know if some of my tests were not run at all because if TeamCity doesn't include then in test results then I don't even know about problems!

Configs: TeamCity 10.0, NUnit 3.0.

Command line params: --result=TestResult.xml --workers=4 --teamcity

Update: results of tests executing in log looks like:

[13:03:48][Step 1/1] Test Run Summary
[13:03:48][Step 1/1]     Overall result: Failed
[13:03:48][Step 1/1]    Tests run: 82, Passed: 0, Errors: 82, Failures: 0, Inconclusive: 0
[13:03:48][Step 1/1]      Not run: 0, Invalid: 0, Ignored: 0, Explicit: 0, Skipped: 0
[13:03:48][Step 1/1]   Start time: 2016-09-08 09:56:33Z
[13:03:48][Step 1/1]     End time: 2016-09-08 10:03:48Z
[13:03:48][Step 1/1]     Duration: 434,948 seconds

So NUnit marks such tests even not as failed but as "erros". Still I want them in test results.

Charlie Charlie · Accepted Answer · 2016-09-08T16:47:36

Your tests are errors because you are throwing an exception in the constructor. Since the test fixture can't be constructed, the test is not really being run as far as NUnit is concerned. The fact that it's an NUnit assertion failure causing the exception is irrelevant in the context of constructing the object.

We have always advised people to keep their constructors very simple because NUnit makes no guarantees about when and how often your object will be constructed. Using assertions in the constructor is an extreme violation of that principal and, in fact, I've never seen anyone do it before.

The OneTimeSetUp attribute is there if you want some thing to happen every time your test is run as opposed to constructed. NUnit does make guarantees about when that method will be executed. :-)

None of this tells me for sure why TC is not recognizing the error but I'm guessing it's because once the constructor fails, the tests are never actually run. NUnit itself compensates for that by reporting the tests as errors but TC would not necessarily do the same.

NUnit tests failed on SetUp are not displayed in TeamCity

1 Answers