What is the correct way to report an error in a Python unittest in the setUp method?

Question

I've read some conflicting advice on the use of assert in the setUp method of a Python unit test. I can't see the harm in failing a test if a precondition that test relies on fails.

For example:

import unittest

class MyProcessor():
    """
    This is the class under test
    """

    def __init__(self):
        pass

    def ProcessData(self, content):
        return ['some','processed','data','from','content'] # Imagine this could actually pass

class Test_test2(unittest.TestCase):

    def LoadContentFromTestFile(self):
        return None # Imagine this is actually doing something that could pass.

    def setUp(self):
        self.content = self.LoadContentFromTestFile()
        self.assertIsNotNone(self.content, "Failed to load test data")
        self.processor = MyProcessor()

    def test_ProcessData(self):
        results = self.processor.ProcessData(self.content)
        self.assertGreater(results, 0, "No results returned")

if __name__ == '__main__':
    unittest.main()

This seems like a reasonable thing to do to me i.e. make sure the test is able to run. When this fails because of the setup condition we get:

F
======================================================================
FAIL: test_ProcessData (__main__.Test_test2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Projects\Experiments\test2.py", line 21, in setUp
    self.assertIsNotNone(self.content, "Failed to load test data")
AssertionError: unexpectedly None : Failed to load test data

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (failures=1)

Answer 1

The purpose of setUp is to reduce Boilerplate code which creates between the tests in the test class during the Arrange phase.

In the Arrange phase you: setup everything needed for the running the tested code. This includes any initialization of dependencies, mocks and data needed for the test to run.

Based on the above paragraphs you should not assert anything in your setUp method.

So as mentioned earlier; If you can't create the test precondition then your test is broken. To avoid situations like this Roy Osherove wrote a great book called: The Art Of Unit Testing ( For a fully disclosure Lior Friedman(He was Roy's boss) is a friend of mine and I worked closely with them for more then 2 years, so I am little bit biased...)

Basically there are only a few reasons to have an interaction with external resources during the Arrange phase(or with things which may cause an exception), most of them(if not all of them) are related in integration tests.

Back to your example; There is a pattern to structure the tests where you need to load an external resource(for all/most of them). Just a side note; before you decide to apply this pattern make sure that you can't has this content as a static resource in your UT's class, if other test classes need to use this resource extract this resource into a module.

The following pattern decrease the possibility for failure, since you have less calls to the external resource:

class TestClass(unittest.TestCase):

    def setUpClass(self):
        # since external resources such as other servers can provide a bad content
        # you can verify that the content is valid
        # then prevent from the tests to run  
        # however, in most cases you shouldn't.
        self.externalResourceContent = loadContentFromExternalResource()


    def setUp(self):
        self.content = self.copyContentForTest()

Pros:

1. less chances to failure
2. prevent inconsistency behavior (1. something/one has edited the external resource. 2. you failed to load the external resource in some of your tests)
3. faster execution

Cons:

1. the code is more complex

Answer 2

setUp is not for asserting preconditions but creating them. If your test is unable to create the necessary fixture, it is broken, not failing.

Answer 3

From the Python Standard Library Documentation:

"If the setUp() method raises an exception while the test is running, the framework will consider the test to have suffered an error, and the runTest() method will not be executed. If setUp() succeeded, the tearDown() method will be run whether runTest() succeeded or not. Such a working environment for the testing code is called a fixture."

An assertion exception in the setUp() method would be considered as an error by the unittest framework. The test will not be executed.

Answer 4

There isn't a right or wrong answer here , it depends on what you are testing and how expensive setting up your tests is. Some tests are too dangerous to allow attempted runs if the data isn't as expected, some need to work with that data.

You can use assertions in setUp if you need to check between tests for particular conditions, this can help reduce repeated code in your tests. However also makes moving test methods between classes or files a bit trickier as they will be reliant on having the equivalent setUp. It can also push the limits of complexity for less code savvy testers.

It a bit cleaner to have a test that checks these startup conditions individually and run it first , they might not be needed in between each test. If you define it as test_01_check_preconditions it will be done before any of the other test methods , even if the rest are random. You can also then use unittest2.skip decorators for certain conditions.

A better approach is to use addCleanup to ensure that state is reset, the advantage here is that even if the test fails it still gets run, you can also make the cleanup more aware of the specific situation as you define it in the context of your test method.

There is also nothing to stop you defining methods to do common checks in the unittest class and calling them in setUp or in test_methods, this can help keep complexity inclosed in defined and managed areas.

Also don't be tempted to subclass unittest2 beyond a simple test definition, i've seen people try to do that to make tests simple and actually introduce totally unexpected behaviour.

I guess the real take home is , if you do it know why you want to use it and ensure you document your reasons it its probably ok , if you are unsure then go for the simplest easiest to understand option because tests are useless if they are not easy to understand.

Answer 5

There is one reason why you want to avoid assertions in a setUp(). If setUp fails, your tearDown will not be executed.

If you setup a set of database records for instance and your teardown deletes these records, then these records will not be deleted.

With this snippet:

import unittest

class Test_test2(unittest.TestCase):

    def setUp(self):
        print 'setup'
        assert False

    def test_ProcessData(self):
        print 'testing'

    def tearDown(self):
        print 'teardown'

if __name__ == '__main__':
    unittest.main()

You run only the setUp():

$ python t.py 
setup
E
======================================================================
ERROR: test_ProcessData (__main__.Test_test2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "t.py", line 7, in setUp
    assert False
AssertionError

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)