Unit testing Anti-patterns catalogue

Question

Locked. This question and its answers are locked because the question is off-topic but has historical significance. It is not currently accepting new answers or interactions.

anti-pattern : there must be at least two key elements present to formally distinguish an actual anti-pattern from a simple bad habit, bad practice, or bad idea:

• Some repeated pattern of action, process or structure that initially appears to be beneficial, but ultimately produces more bad consequences than beneficial results, and
• A refactored solution that is clearly documented, proven in actual practice and repeatable.

Vote for the TDD anti-pattern that you have seen "in the wild" one time too many.
The blog post by James Carr and Related discussion on testdrivendevelopment yahoogroup

If you've found an 'unnamed' one.. post 'em too. One post per anti-pattern please to make the votes count for something.

My vested interest is to find the top-n subset so that I can discuss 'em in a lunchbox meet in the near future.

Answer 1

Second Class Citizens - test code isn't as well refactored as production code, containing a lot of duplicated code, making it hard to maintain tests.

Answer 2

The Free Ride / Piggyback -- James Carr, Tim Ottinger
Rather than write a new test case method to test another/distinct feature/functionality, a new assertion (and its corresponding actions i.e. Act steps from AAA) rides along in an existing test case.

Answer 3

Happy Path

The test stays on happy paths (i.e. expected results) without testing for boundaries and exceptions.

JUnit Antipatterns

Answer 4

The Local Hero

A test case that is dependent on something specific to the development environment it was written on in order to run. The result is the test passes on development boxes, but fails when someone attempts to run it elsewhere.

The Hidden Dependency

Closely related to the local hero, a unit test that requires some existing data to have been populated somewhere before the test runs. If that data wasn’t populated, the test will fail and leave little indication to the developer what it wanted, or why… forcing them to dig through acres of code to find out where the data it was using was supposed to come from.

---

Sadly seen this far too many times with ancient .dlls which depend on nebulous and varied .ini files which are constantly out of sync on any given production system, let alone extant on your machine without extensive consultation with the three developers responsible for those dlls. Sigh.

Answer 5

Chain Gang

A couple of tests that must run in a certain order, i.e. one test changes the global state of the system (global variables, data in the database) and the next test(s) depends on it.

You often see this in database tests. Instead of doing a rollback in teardown(), tests commit their changes to the database. Another common cause is that changes to the global state aren't wrapped in try/finally blocks which clean up should the test fail.

Answer 6

The Mockery
Sometimes mocking can be good, and handy. But sometimes developers can lose themselves and in their effort to mock out what isn’t being tested. In this case, a unit test contains so many mocks, stubs, and/or fakes that the system under test isn’t even being tested at all, instead data returned from mocks is what is being tested.

Source: James Carr's post.

Answer 7

The Silent Catcher -- Kelly?
A test that passes if an exception is thrown.. even if the exception that actually occurs is one that is different than the one the developer intended.
See Also: Secret Catcher

[Test]
[ExpectedException(typeof(Exception))]
public void ItShouldThrowDivideByZeroException()
{
   // some code that throws another exception yet passes the test
}

Answer 8

The Inspector
A unit test that violates encapsulation in an effort to achieve 100% code coverage, but knows so much about what is going on in the object that any attempt to refactor will break the existing test and require any change to be reflected in the unit test.

---

'how do I test my member variables without making them public... just for unit-testing?'

Answer 9

Excessive Setup -- James Carr
A test that requires a huge setup in order to even begin testing. Sometimes several hundred lines of code are used to prepare the environment for one test, with several objects involved, which can make it difficult to really ascertain what is tested due to the “noise” of all of the setup going on. (Src: James Carr's post)

Answer 10

Anal Probe

A test which has to use insane, illegal or otherwise unhealthy ways to perform its task like: Reading private fields using Java's setAccessible(true) or extending a class to access protected fields/methods or having to put the test in a certain package to access package global fields/methods.

If you see this pattern, the classes under test use too much data hiding.

The difference between this and The Inspector is that the class under test tries to hide even the things you need to test. So your goal is not to achieve 100% test coverage but to be able to test anything at all. Think of a class that has only private fields, a run() method without arguments and no getters at all. There is no way to test this without breaking the rules.

---

Comment by Michael Borgwardt: This is not really a test antipattern, it's pragmatism to deal with deficiencies in the code being tested. Of course it's better to fix those deficiencies, but that may not be possible in the case of 3rd party libraries.

Aaron Digulla: I kind of agree. Maybe this entry is really better suited for a "JUnit HOWTO" wiki and not an antipattern. Comments?

Answer 11

The Test With No Name -- Nick Pellow

The test that gets added to reproduce a specific bug in the bug tracker and whose author thinks does not warrant a name of its own. Instead of enhancing an existing, lacking test, a new test is created called testForBUG123.

Two years later, when that test fails, you may need to first try and find BUG-123 in your bug tracker to figure out the test's intent.

Answer 12

The Slow Poke

A unit test that runs incredibly slow. When developers kick it off, they have time to go to the bathroom, grab a smoke, or worse, kick the test off before they go home at the end of the day. (Src: James Carr's post)

a.k.a. the tests that won't get run as frequently as they should

Answer 13

The Butterfly

You have to test something which contains data that changes all the time, like a structure which contains the current date, and there is no way to nail the result down to a fixed value. The ugly part is that you don't care about this value at all. It just makes your test more complicated without adding any value.

The bat of its wing can cause a hurricane on the other side of the world. -- Edward Lorenz, The Butterfly Effect

Answer 14

The Flickering Test (Source : Romilly Cocking)

A test which just occasionally fails, not at specific times, and is generally due to race conditions within the test. Typically occurs when testing something that is asynchronous, such as JMS.

Possibly a super set to the 'Wait and See' anti-pattern and 'The Sleeper' anti-pattern.

The build failed, oh well, just run the build again. -- Anonymous Developer

Answer 15

Wait and See

A test that runs some set up code and then needs to 'wait' a specific amount of time before it can 'see' if the code under test functioned as expected. A testMethod that uses Thread.sleep() or equivalent is most certainly a "Wait and See" test.

Typically, you may see this if the test is testing code which generates an event external to the system such as an email, an http request or writes a file to disk.

Such a test may also be a Local Hero since it will FAIL when run on a slower box or an overloaded CI server.

The Wait and See anti-pattern is not to be confused with The Sleeper.

Answer 16

Inappropriately Shared Fixture -- Tim Ottinger
Several test cases in the test fixture do not even use or need the setup / teardown. Partly due to developer inertia to create a new test fixture... easier to just add one more test case to the pile

Answer 17

The Giant

A unit test that, although it is validly testing the object under test, can span thousands of lines and contain many many test cases. This can be an indicator that the system under tests is a God Object (James Carr's post).

A sure sign for this one is a test that spans more than a a few lines of code. Often, the test is so complicated that it starts to contain bugs of its own or flaky behavior.

Answer 18

I'll believe it when I see some flashing GUIs
An unhealthy fixation/obsession with testing the app via its GUI 'just like a real user'

Testing business rules through the GUI is a terrible form of coupling. If you write thousands of tests through the GUI, and then change your GUI, thousands of tests break.
Rather, test only GUI things through the GUI, and couple the GUI to a dummy system instead of the real system, when you run those tests. Test business rules through an API that doesn't involve the GUI. -- Bob Martin

“You must understand that seeing is believing, but also know that believing is seeing.” -- Denis Waitley

Answer 19

The Sleeper, aka Mount Vesuvius -- Nick Pellow

A test that is destined to FAIL at some specific time and date in the future. This often is caused by incorrect bounds checking when testing code which uses a Date or Calendar object. Sometimes, the test may fail if run at a very specific time of day, such as midnight.

'The Sleeper' is not to be confused with the 'Wait And See' anti-pattern.

That code will have been replaced long before the year 2000 -- Many developers in 1960

Answer 20

The Dead Tree

A test which where a stub was created, but the test wasn't actually written.

I have actually seen this in our production code:

class TD_SomeClass {
  public void testAdd() {
    assertEquals(1+1, 2);
  }
}

I don't even know what to think about that.

Answer 21

got bit by this today:

Wet Floor:
The test creates data that is persisted somewhere, but the test does not clean up when finished. This causes tests (the same test, or possibly other tests) to fail on subsequent test runs.

In our case, the test left a file lying around in the "temp" dir, with permissions from the user that ran the test the first time. When a different user tried to test on the same machine: boom. In the comments on James Carr's site, Joakim Ohlrogge referred to this as the "Sloppy Worker", and it was part of the inspiration for "Generous Leftovers". I like my name for it better (less insulting, more familiar).

Answer 22

The Cuckoo -- Frank Carver
A unit test which sits in a test case with several others, and enjoys the same (potentially lengthy) setup process as the other tests in the test case, but then discards some or all of the artifacts from the setup and creates its own.
Advanced Symptom of : Inappropriately Shared Fixture

Answer 23

The Secret Catcher -- Frank Carver
A test that at first glance appears to be doing no testing, due to absence of assertions. But "The devil is in the details".. the test is really relying on an exception to be thrown and expecting the testing framework to capture the exception and report it to the user as a failure.

[Test]
public void ShouldNotThrow()
{
   DoSomethingThatShouldNotThrowAnException();
}

Answer 24

The Environmental Vandal

A 'unit' test which for various 'requirements' starts spilling out into its environment, using and setting environment variables / ports. Running two of these tests simultaneously will cause 'unavailable port' exceptions etc.

These tests will be intermittent, and leave developers saying things like 'just run it again'.

One solution Ive seen is to randomly select a port number to use. This reduces the possibility of a conflict, but clearly doesnt solve the problem. So if you can, always mock the code so that it doesn't actually allocate the unsharable resource.

Answer 25

The Turing Test

A testcase automagically generated by some expensive tool that has many, many asserts gleaned from the class under test using some too-clever-by-half data flow analysis. Lulls developers into a false sense of confidence that their code is well tested, absolving them from the responsibility of designing and maintaining high quality tests. If the machine can write the tests for you, why can't it pull its finger out and write the app itself!

Hello stupid. -- World's smartest computer to new apprentice (from an old Amiga comic).

Answer 26

The Forty Foot Pole Test

Afraid of getting too close to the class they are trying to test, these tests act at a distance, separated by countless layers of abstraction and thousands of lines of code from the logic they are checking. As such they are extremely brittle, and susceptible to all sorts of side-effects that happen on the epic journey to and from the class of interest.

Answer 27

Doppelgänger

In order to test something, you have to copy parts of the code under test into a new class with the same name and package and you have to use classpath magic or a custom classloader to make sure it is visible first (so your copy is picked up).

This pattern indicates an unhealthy amount of hidden dependencies which you can't control from a test.

I looked at his face ... my face! It was like a mirror but made my blood freeze.

Answer 28

The Mother Hen -- Frank Carver
A common setup which does far more than the actual test cases need. For example creating all sorts of complex data structures populated with apparently important and unique values when the tests only assert for presence or absence of something.
Advanced Symptom of: Inappropriately Shared Fixture

I don't know what it does ... I'm adding it anyway, just in case. -- Anonymous Developer

Answer 29

The Test It All

I can't believe this hasn't been mentioned till now, but tests should not break the Single Responsibility Principle.

I have come across this so many times, tests that break this rule are by definition a nightmare to maintain.

Answer 30

Line hitter

On the first look tests covers everything and code coverage tools confirms it with 100%, but in reality tests only hit code without any output analyses.

coverage-vs-reachable-code