PHPUnit: Multiple assertions in a single test, only first failure seen

Question

The next weirdness I'm seeing with PHPUnit:

class DummyTest extends PHPUnit_Framework_TestCase {
    public function testDummy() {
        $this->assertTrue(false, 'assert1');
        $this->assertTrue(false, 'assert2');
    }

    public function testDummy2() {
        $this->assertTrue(false, 'assert3');
    }
}

As soon as the first assertion fails in a test, the rest of the test is ignored.

So (with a simple call of phpunit DummyTest.php):

• The above code will display 2 tests, 2 assertions, 2 failures. What?

• If I make all the tests pass, then I'll get OK (2 tests, 3 assertions). Good.

• If I only make all the tests pass except for assert2, I get 2 tests, 3 assertions, 1 failure. Good.

I don't get it, but PHPUnit's been around for ages, surely it has to be me?

Not only are the counts not what I'd expect, only the error message for the first failed assert in the code above is displayed.

(BTW, I'm analyzing the xml format generated by PHPUnit for CI rather than testing real code, hence the practice of multiple assertions in the one test.)

Answer 1

First off: That is expected behavior.

Every test method will stop executing once an assertion fails.

For an example where the opposite will be very annoying*:

class DummyTest extends PHPUnit_Framework_TestCase {
    public function testDummy() {
        $foo = get_me_my_foo();
        $this->assertInstanceOf("MyObject", $foo);
        $this->assertTrue($foo->doStuff());
    } 
}

if phpunit wouldn't stop after the first assertion you'd get an E_FATAL (call to a non member function) and the whole PHP process would die.

So to formulate nice assertions and small tests it's more practical that way.

For another example:

When "asserting that an array has a size of X, then asserting that it contains a,b and c" you don't care about the fact that it doesn't contain those values if the size is 0.

If a test fails you usually just need the message "why it failed" and then, while fixing it, you'll automatically make sure the other assertions also pass.

---

On an additional note there are also people arguing that you should only have One Asssertion per Test case while I don't practice (and I'm not sure if i like it) I wanted to note it ;)

Answer 2

Welcome to unit testing. Each test function should test one element or process (process being a series of actions that a user might take). (This is a unit, and why it is called "unit testing.") The only time you should have multiple assertions in a test function is if part of the test is dependent on the previous part being successful.

I use this for Selenium testing web pages. So, I might want to assert that I am in the right place every time I navigate to a new page. For instance, if I go to a web page, then login, then change my profile, I would assert that I got to the right place when I logged in, because the test would no longer make sense if my login failed. This prevents me from getting additional error messages, when only one problem was actually encountered.

On the other side, if I have two separate processes to test, I would not test one, then continue on to test the other in the same function, because an error in the first process would mask any problems in the second. Instead, I would write one test function for each process. (And, if one process depended on the success of the other, for instance, post something to a page, then remove the post, I would use the @depends annotation to prevent the second test from running if the first fails.)

In short, if your first assert failing does not make the second one impossible to test, then they should be in separate functions. (Yes, this might result in redundant code. When unit testing, forget all that you have learned about eliminating redundant code. That, or make non-test functions and call them from the test functions. This can make unit tests harder to read, and thus harder to update when changes are made to the subject of the tests though.)

I realize that this question is 2 years old, however the only answer was not very clear about why. I hope that this helps others understand unit testing better.