Boundaries of acceptance tests

Question

my application, among other things, uses some crawlers to read information exposed by a remote xml feed by another application (we're not responsible for this one). Crawled data is later displayed to the user. The xml might contain simple data and links, that we follow if we need additional data.

The tests in our system are both unit tests, that test that we parse correctly the xml documents, and acceptance tests, that are meant to test what we display in our ui.

I was reasoning about the acceptance tests, and that's what this question is about. Right now, for each acceptance test, we bring an embedded http server that serves some test data, that is specific for the test. We then start up our application, we crawl the test data and we verify the criteria for the test. While this approach has the advantage of testing the whole system from end to end, it has also the side effect of increasing the build time considerably each time we add a new acceptance test.

Is this the right approach for the acceptance tests? I was wondering if, since the system that provides the feeds is an external one, wouldn't it be better to test the network communication layer and the crawlers at unit level and run the acceptance tests assuming the data has already been crawled?

I'd like to hear some thought from somebody else. :-)

Thanks!

Answer 1

Acceptance tests do tend to run slowly, require more setup and tend to be far more brittle than unit or integration tests. If you search the web for "test pyramid" you will find plenty of information on this. The general consensus is that you should have tests at the unit, integration and acceptance levels. With most tests being unit tests and just a few acceptance tests that do the end-to-end stuff. Often development teams will setup their ci servers to only run any long running acceptance tests during their nightly build processes so that they don't impact the performance of the unit test test runs.

Answer 2

I agree with what Andrew wrote, but wanted to add a different angle to the answer, one which I think is quite often missed in such discussions.

Your team is building a product and your company wants to get the best value for money from this endeavor.

At the beginning you might think that tests slow you down - your system is simple and everyone understands it so why waste time. It might feel like you get little value for money from writing tests. But this is obviously wrong if you adopt a longer term view of your product development. But I'll stop here, as I'm preaching to the converted.

However, if you adopt the same mindset when trying to answer your question, you will see that actually the answer depends a lot on your circumstances. I'm going to use a rather simplified math model to explain my thinking:

Let P(bug | test) denote a probability of a bug, provided you are running the test, let C(test) denote the cost of running the test and let C(bug) denote the cost of a bug.

If you focus on a particular bug*, you want to minimize the following:

P(bug | test_1)*C(bug) + C(test_1) ... P(bug | test_n)*C(bug) + C(test_n)

Where your suite consists of n tests.

If you were to disregard the test cost, clearly the more tests you have, the better right? But because tests need to be maintained, executed, etc., they have a non-zero cost. This means that you have a trade-off and in the end you are performing a U-curve optimization here (a bit like on this picture where they are trying to find the optimal tradeoff between release and holding costs).

The actual costs depend a lot on a particular domain, product areas and test types.

If you are in banking the cost of a bug can be enormous, so it will dwarf the test costs. But if you are writing a recommendation engine for music, having suggestions that are off for a few hours will not be a problem. Actually, in the latter case, you probably want the freedom to experiment with different algorithms and the ability to iterate quickly, so the cost of a test might over-shaddow the cost of a bug.

Lets say you work on a particular product. Even that is not homogenous. There will be areas of your product that are more critical than others. Take twitter for example, if a person could not tweet or load the tweets of who they follow it would be a big problem. On the other hand, if "who to follow suggestions" are empty, the impact on the product will be much smaller.

Finally, the cost of tests is not uniform either. But as I said earlier, it is not negligible and needs to be considered with care. I worked both in places where poor test coverage slowed the teams down because they lacked the confidence to push their changes to production, and in places where test runs were so long that people complained that they are constantly building and hardly working.

One last thing. It is good to build with resiliency to failure in mind - will lower the cost of bugs for you.