The number of unit testing combinations is staggering

Question

I'm new to unit testing (recent grad now in the real world). I'm baffled at how long it takes to write tests.

Let's take a basic example. I write a function like:

def add(a, b)
    return a + b

and I want to test it for inputs of integers and floats of unlimited precision. For example, a test case name may be test_add_negative_integer_to_negative_integer.

Assumption
Testing edge cases at the boundaries is representative of all other cases.
Edges/Boundaries
Numerical types = {integer, float}
Numerical values = {negative, zero, positive}
Number of test cases
Combinations with repetition (Assuming parameter order is not important).

C(3 + 2 - 1, 2) * C(2 + 2 - 1, 1) = 18 test cases to meet the assumption condition.
Adding another value to to the numerical types set yields 36 different test cases.

Am I doing something wrong?

Answer 1

What is the goal of your test?

Is it an exhaustive coverage of all possible inputs? It would take 18 test cases to fully cover all combinations, as you listed, but are those combinations necessary??

A risk analysis goes hand in hand with determining how exhaustive tests have to be for code. What is both the technical risk and the business risk associated with this method having defects? If it is a critical risk then all 18 combinations are probably necessary. In addition some sort of random testing is very helpful as well.

IMO, for unit testing, I have found that coverage is a good guiding metric, that provides a high degree of confidence, while avoiding all combinations. If you use coverage as a goal, then this method would only require a single unit test.

If this is a critical function, and requires all combinations, I would recommend a data-driven approach. It is a clean way to specify all 18 test cases and drive each test using the same code. It solves the problem of having to write 18 separate test_this_and_this.

Using nose-parameterized:

@parameterized([
    (1, -1.0, 0),
    (2, 3, 8),
    (1, 9, 1),
    (0, 9, 0),
])
def test_add(first, second, expected):
    assert_equal(add(first, second), expected)

Data driven tests are also a wonderful organizational tool. If you find yourself working with non-technical testers, it is powerful to be able to create a test driver, then have someone non-technical design a CSV with all the relevant business related test cases, and provide it as input to your data driven test.

Answer 2

I think the answer to your question is simple: apply the TDD process, and you'll see the problem automatically goes away.

To be more specific, you would start by writing your first test for the "add(a, b)" feature. This can be done with a single-line test method, for an initially empty implementation of the "add" method. The test (say add(1, 2) == 3) will fail as expected.

After getting this first test to pass, and only then, you would try and write a second test.

And here is the key to why this process works: the second test first needs to fail for a valid reason. The "valid reason" would be directly related to some other business requirement that the current "add" implementation does not yet fulfill. If you can't find such a requirement, then there won't be a second test.

Repeat the above until no new tests can be added. You'll see that a much smaller number of tests will be needed to achieve full coverage of the desired functionality.

Answer 3

Unit testing is a practical way of writing/testing code (TDD is writing your unit test first to help you develop your code).

A couple of things you should consider:

• Test only your code. Testing others' API (in your case, the system's API) is a sink hole just like you described (it is no longer unit testing). This would only require one test: verify when calling add + is called.

• Test only that which could possibly break. Getters and setters and simple wrappers (like you have here) are not likely to break by you or one of your teammates. Bugs occur in places (ie., cyclomatic complexity, strange coupling, code with lots of churn), this is where you want most of your tests.

• Finally, if you were writing the code for system +, you might want all those 18 tests but probably not, there is a certain symmetry to integers.

Remember, unit testing is to help you develop and maintain code. If it is not helping, try doing less.

I usually have 1 1/2 times as much unit test code as I do production code.