Automated acceptance testing - UI or API?

Question

I've been researching automated acceptance testing over the last few days, learning about BDD & JBehave, FitNesse & Slim, Selenium & WebDriver, etc.

I've just viewed this video by Robert C. Martin where he demonstrates how to use FitNesse to write and maintain such tests. Towards the end, somebody asks if these tests hit the UI. Martin goes on to explain that coupling acceptance tests to the UI can be costly since changes to the UI are quite frequent. I could guess also that such tests could only be written after the UI has been developed, which would bring testers behind schedule by definition.

I have to ask: what is the alternative? Martin seems to be implying that tests should be hitting a hidden layer that would manipulate the application's business layer. My understanding is that this would require additional work, not to mention that it would expose a new API which would need to be secured once in a production environment.

Could hitting the business layer through application services be sufficient?

What has been your experience?

Thanks for sharing!

Answer 1

Testing through the UI or hitting the business layer directly can be seen as two different types of tests, with different advantages and disadvantages.

If you're testing the UI directly, then you're testing what the user sees, and you don't have to change code in order to be able to test it. However, it becomes quite hard to test corner cases, or how the system reacts to exceptional conditions (such as exceptions). It is as Robert Martin says, brittle. If your interface changes, you need to change your tests. So testing through the UI depends upon the maturity of your UI. Later on in the project, the UI is more stable, so testing through the UI makes more sense. Also, testing some stuff through the UI is hard or convoluted.

If you're testing the business layer, you can then more easily test corner conditions, and you are less susceptible to changes in the UI. However, as you say, the software has to be written in such a way as to allow testing of this kind. You may even have to expose a new interface to allow it, which then has to be maintained. In my experience, it is sometimes quite hard to get the developers to support this sort of interface. It's seen as not as important as the real interface. But it's always possible. This sort of interface needs buy-in from the developers, otherwise you risk it being 'not supported' over time.

If you don't have the external interface, then try asking for it. You never know, you might be able to persuade them that it's a good idea. It's easier at the start of a project.

If you have the external interface, then use that to test your business logic.

Otherwise, you'll have to use the UI to test these things. One approach would be to use the UI to do smoke testing, to answer the following question(s): Is this software testable? Think of the common things you have to test every time you get a build from dev. Can I login, can I logout, does the main page appear, can I do a simple order? Pick 5 or 6 of these things, and build an automated test suite to test these things. Use these tests as a guide as to how much functionality you can actually test through the UI, and how useful it is.

You can then use this as an argument when you go to the developers and ask for the external interface.

Answer 2

Unfortunately you need both. In general you would want to automate the business layer tests with something like fitness. Avoiding the UI basically gives you assurance that the defined business rules always work. Automating thru the UI can require a lot more maintenance. But on the plus side since much of the UI uses mechanisms provided by the platform (e.g. c#/Winforms) the bulk of your UI tests can be only when changes are made if you are well covered with other types of tests (like business layer). Automated tests need to be maintained and updated but UI tests tend to require more maintenance and are often less reliable over time.

Answer 3

I strongly advocate using non-UI mode of automation for acceptance tests. The problem you may face is in selling the idea to traditional code-blind testers who would find the concept of a non visual verification of an acceptance criteria inadequate. I prefer a mid way where an acceptance criteria is tested by two sets of tests. The first set is purely logic focused and hence implemented against an API/Service/Remote layer. This set is 100% automated. The second set is a UI verification of the same requirement. My personal opinion is not to automate the second set for reasons stated in other answers. As for a layer to write the tests against, it depends on the team. If the team is truly committed towards quality, they will see this as an essential part of product development. However, in real world, it is a hard sell. Unless it is explicitly called out from the starting of a project, it would be difficult to retrofit such an interface in a running project. I would suggest, if you want to implement automated acceptance tests, do it as early as possible. As the product ages, the likelihood of such hacks being implemented decreases drastically.

Answer 4

I can only speak for UI testing, Q&A style:

A) Is our UI already stable or still likely to change quite a lot?

Stable - automation brings extra value

Likely to change - don't automate, you will spend more time maintaining (redoing) these tests.

---

B) Our UI is stable, should we automate as much of it as possible?

God No!

1) Take all your test scripts (say, 100)

2) Decide which are automatable (80)

3) Sort them by priority (blockers, critical, minor)

4) Automate the blockers and critical ones first (say, 20/80)

See how that works, and automate more if need arises.

---

C) What other factors should I consider when automating UI?

Everything as shown below (test pyramid is a concept developed by Mike Cohn):

Answer 5

Creating an interface for testing is "the dream of testers" but enabling / disabling it will be the nightmare for deployment. U may need "test builds" and "deploy builds" with different smoke tests. But any compromise is good as long u limit UI automate testing.

Answer 6

As far as Mobile Automation is concerned, I believe we don't need separate interface for non-ui tests. These days there are ways to write code using patterns such that the code is unit testable (e.g. Dependency Injection & a way to provide even the private methods available in tests. (e.g. @VisibleForTesting).

Note: I haven't ever written non-ui tests yet nor I have ever used this annotation. , but looking forward to it. I just wanted to share the @VisibleForTesting thing to break the notion that we need separate methods as an interface for testing purposes.) Some one's post: https://medium.com/@vadimpushtaev/how-to-test-private-methods-4bc57d4410ff