Why using Integration tests instead of unit tests is a bad idea?

Question

Let me start from definition:

Unit Test is a software verification and validation method in which a programmer tests if individual units of source code are fit for use

Integration testing is the activity of software testing in which individual software modules are combined and tested as a group.

Although they serve different purposes very often these terms are mixed up. Developers refer to automated integration tests as unit tests. Also some argue which one is better which seems to me as a wrong question at all.

I would like to ask development community to share their opinions on why automated integration tests cannot replace classic unit tests.

Here are my own observations:

1. Integration tests can not be used with TDD approach
2. Integration tests are slow and can not be executed very often
3. In most cases integration tests do not indicate the source of the problem
4. it's more difficult to create test environment with integration tests
5. it's more difficult to ensure high coverage (e.g. simulating special cases, unexpected failures etc)
6. Integration tests can not be used with Interaction based testing
7. Integration tests move moment of discovering defect further (from paxdiablo)

EDIT: Just to clarify once again: the question is not about whether to use integration or unit testing and not about which one is more useful. Basically I want to collect arguments to the development teams which write ONLY integration tests and consider them as unit tests. Any test which involve components from different layers is considered as integration test. This is to compare to unit test where isolation is the main goal.

Thank you, Andrey

Answer 1

Integration tests tell you whether it's working. Unit tests tell you what isn't working. So long as everything is working, you "don't need" the unit tests - but once something is wrong, it's very nice to have the unit test point you directly to the problem. As you say, they serve different purposes; it's good to have both.

To directly address your subject: integration tests aren't a problem, aren't the problem. Using them instead of unit tests is.

Answer 2

There have been studies^(a) that show that the cost of fixing a bug becomes higher as you move away from the point where the bug was introduced.

For example, it will generally cost you relatively little to fix a bug in software you haven't even pushed up to source control yet. It's your time and not much of it, I'd warrant (assuming you're any good at your job).

Contrast that with how much it costs to fix when the customer (or all your customers) find that problem. Many level of people get involved and new software has to be built in a hurry and pushed out to the field.

That's the extreme comparison. But even the difference between unit and integration tests can be apparent. Code that fails unit testing mostly affects only the single developer (unless other developers/testers/etc are waiting on it, of course). However, once your code becomes involved in integration testing, a defect can begin holding up other people on your team.

We wouldn't dream of replacing our unit tests with integration tests since:

• Our unit tests are automated as well so, other than initial set-up, the cost of running them is small.
• They form the beginning of the integration tests. All unit tests are rerun in the integration phase to check that the integration itself hasn't broken anything, and then there are the extra tests that have been added by the integration team.

---

^(a) See, for example, http://slideshare.net/Vamsipothuri/defect-prevention, slide # 5, or search the net for Defect prevention : Reducing costs and enhancing quality. Th graph from the chart is duplicated below in case it ever becomes hard to find on the net:

Answer 3

I find integration tests markedly superior to unit tests. If I unit test my code, I'm only testing what it does versus my understanding of what it should do. That only catches implementation errors. But often a much bigger problem is errors of understanding. Integration tests catch both.

In addition, there is a dramatic cost difference; if you're making intensive use of unit tests, it's not uncommon for them to outweigh all the rest of your code put together. And they need to be maintained, just like the rest of the code does. Integration tests are vastly cheaper -- and in most cases, you already need them anyway.

There are rare cases where it might be necessary to use unit tests, e.g. for internal error handling paths that can't be triggered if the rest of the system is working correctly, but most of the time, integration tests alone give better results for far lower cost.

Answer 4

In many cases you need both. Your observations are right on track as far as I'm concerned with respect to using integration tests as unit tests, but they don't mean that integration tests are not valuable or needed, just that they serve a different purpose. One could equally argue that unit tests can't replace integration tests, precisely because they remove the dependencies between objects and they don't exercise the real environment. Both are correct.

Answer 5

• Integration tests are slow.
• Integration tests may break different reasons (it is not focused and isolated). Therefore you need more debugging on failures.
• Combination of scenarios are to big for integration test when it is not unit tested.

Mostly I do unit tests and 10 times less integration tests (configuration, queries).

Answer 6

It's all about reducing the iteration time.

With unit tests, you can write a line of code and verify it in a minute or so. With integration tests, it usually takes significantly longer (and the cost increases as the project grows).

Both are clearly useful, as both will detect issues that the other fails to detect.

OTOH, from a "pure" TDD approach, unit tests aren't tests, they're specifications of functionality. Integration tests, OTOH, really do "test" in the more traditional sense of the word.

Answer 7

The two types of tests are different. Unit tests, in my opinion are not a alternative to integration tests. Mainly because integration tests are usually context specific. You may well have a scenario where a unit test fails and your integration doesn't and vice versa. If you implement incorrect business logic in a class that utilizes many other components, you would want your integration tests to highlight these, your unit tests are oblivious to this.I understand that integration testing is quick and easy. I would argue you rely on your unit tests each time you make a change to your code-base and having a list of greens would give you more confidence that you have not broken any expected behavior at the individual class level. Unit tests give you a test against a single class is doing what it was designed to do. Integration tests test that a number of classes working together do what you expect them to do for that particular collaboration instance. That is the whole idea of OO development: individual classes that encapsulate particular logic, which allows for reuse.

Answer 8

Integration testing generally happens after unit testing. I'm not sure what value there is in testing interactions between units that have not themselves been tested.

There's no sense in testing how the gears of a machine turn together if the gears might be broken.

Answer 9

I think coverage is the main issue.

A unit test of a specific small component such as a method or at most a class is supposed to test that component in every legal scenario (of course, one abstracts equivalence classes but every major one should be covered). As a result, a change that breaks the established specification should be caught at this point.

In most cases, an integration uses only a subset of the possible scenarios for each subunit, so it is possible for malfunctioning units to still produce a program that initially integrates well.

It is typically difficult to achieve maximal coverage on the integration testing for all the reasons you specified below. Without unit tests, it is more likely that a change to a unit that essentially operates it in a new scenario would not be caught and might be missed in the integration testing. Even if it is not missed, pinpointing the problem may be extremely difficult.

I am not sure that most developers refer to unit tests as integration tests. My impression is that most developers understand the differences, which does not mean they practice either.

Answer 10

A unit test is written to test a method on a class. If that class depends on any kind of external resource or behavior, you should mock them, to ensure you test just your single class. There should be no external resources in a unit test.

An integration test is a higher level of granularity, and as you stated, you should test multiple components to check if they work together as expected. You need both integration tests and unit tests for most projects. But it is important they are kept separate and the difference is understood.

Unit tests, in my opinion, are more difficult for people to grasp. It requires a good knowledge of OO principles (fundamentally based on one class one responsibility). If you are able to test all your classes in isolation, chances are you have a well design solution which is maintainable, flexible and extendable.

• When you check-in, your build server should only run unit tests and they should be done in a few seconds, not minutes or hours.
• Integration tests should be ran overnight or manually as needed.

Answer 11

• Unit tests focus on testing an individual component and do not rely on external dependencies. They are commonly used with mocks or stubs.
• Integration tests involve multiple components and may rely on external dependencies.

I think both are valuable and neither one can replace the other in the job they do. I do see a lot of integration tests masquerading as unit tests though having dependencies and taking a long time to run. They should function separately and as part of a continuous integration system.

Integration tests do often find things that unit tests do not though...

Answer 12

Integration tests let you check that whole use cases of your application work.

Unit tests check that low-level logic in your application is correct.

Integration tests are more useful for managers to feel safer about the state of the project (but useful for developers too!).

Unit tests are more useful for developers writing and changing application logic.

And of course, use them both to achieve best results.

Answer 13

It is a bad idea to "use integration tests instead of unit tests" because it means you aren't appreciating that they are testing different things, and of course passing and failing tests will give you different information. They make up sort of a ying and yang of testing as they approach it from either side.

• Integration tests take an approach that simulates how a user would interact with the application. These will cut down on the need for as much manual testing, and passing tests will can tell you that you app is good to go on multiple platforms. A failing test will tell you that something is broken but often doesn't give you a whole lot of information about what's wrong with the underlying code.

• Unit tests should be focusing on making sure the inputs and outputs of your function are what you expect them to be in all cases. Passing units tests can mean that your functions are working according to spec (assuming you have tests for all situations). However, all your functions working properly in isolation doesn't necessarily mean that everything will work perfectly when it's deployed. A failing unit test will give you detailed, specific information about why it's failing which should in theory make it easier to debug.

In the end I believe a combination of both unit and integration tests will yield the quickest a most bug-free software. You could choose to use one and not the other, but I avoid using the phrase "instead of".

Answer 14

How I see integration testing & unit testing:

Unit Testing: Test small things in isolation with low level details including but not limited to 'method conditions', checks, loops, defaulting, calculations etc.

Integration testing: Test wider scope which involves number of components, which can impact the behaviour of other things when married together. Integration tests should cover end to end integration & behaviours. The purpose of integration tests should be to prove systems/components work fine when integrated together.

Answer 15

(I think) What is referred here by OP as integration tests are leaning more to scenario level tests.

But where do we draw the line between unit -> integration -> scenario?

What I often see is developers writing a feature and then when unit testing it mocking away every other piece of code this feature uses/consumes and only test their own feature-code because they think someone else tested that so it should be fine. This helps code coverage but can harm the application in general.

In theory the small isolation of Unit Test should cover a lot since everything is tested in its own scope. But such tests are flawed and do not see the complete picture.

A good Unit test should try to mock as least as possible. Mocking API and persistency would be something for example. Even if the application itself does not use IOC (Inversion Of Control) it should be easy to spin up some objects for a test without mocking if every developer working on the project does it as well it gets even easier. Then the test are useful. These kind of tests have an integration character to them aren't as easy to write but help you find design flaws of your code. If it is not easy to test then adapt your code to make it easy to test. (TDD)

Pros

1. Fast issue identification
2. Helps even before a PR merge
3. Simple to implement and maintain
4. Providing a lot of data for code quality checking (e.g. coverage etc.)
5. Allows TDD (Test Driven Development)

Cons

1. Misses scenario integration errors
2. Succumbs to developer blindness in their own code(happens to all of us)

A good integration test would be executed for complete end to end scenarios and even check persistency and APIs which the unit test could not cover so you might know where to look first when those fail.

Pros:

1. Test close to real world e2e scenario
2. Finds Issues that developers did not think about
3. Very helpful in microservices architectures

Cons:

1. Most of the time slow
2. Need often a rather complex setup
3. Environment (persistency and api) pollution issues (needs cleanup steps)
4. Mostly not feasible to be used on PR's (Pull Requests)

TLDR: You need both you cant replace one with the other! The question is how to design such tests to get the best from both. And not just have them to show good statistics to the management.