Unit testing for a Selenium project

Question

A question about best practices or practice at all ;)

I am currently working on a test automation system using Selenium in Java. It is supposed to be used for end-to-end acceptance testing of a webapp. The test cases are written in the Gherkin language and executed by the BDD framework Cucumber (Cucumber-JVM). The low-level functions use Selenium/WebDriver for interacting with the AUT and the browser. The Selenium code is structured using the PageObject pattern which abstracts the usage of WebDriver away. The cucumber step definitions call just the methods provided by the PageObjects.

As the project continues and becomes more and more complex I would like to start writing unit test to make sure the acceptance tests, and the utility functions around those, do what they should :)

Now to the question:

Is it feasible to write unit test for testing a test automation project?

The main problem is that during my first approach to unit testing using TestNG I realised, that my unit tests ended up doing more or less the same stuff the acceptance tests already did. This is counter productive, as the unit tests are very slow and have a lot dependencies.

Or does one just test the utility classes and leave the Selenium code be in such a case. ie. test just the stuff that can be tested without calling the Selenium WebDriver and interacting with the AUT?

Note just to be sure I have not been misunderstood. I'm asking about running unit test ON the acceptance test code and all the auxiliary code. Not about running the Selenium test cases using an unit testing framework like JUnit or TestNG.

Any help and/or ideas will be appreciated, as I am not sure how to tackle this one. That is if writing tests for tests is at all sensible ;)

Answer 1

I'm sure someone will consider my response "opinionated" and vote it down, but nevertheless I think that

Yes, if you have a test framework you are relying upon for your acceptance testing, the framework itself needs to be tested.

From my experience the value is in 2 areas:

1. Be able to change your framework with confidence. When you create some function, you know quite a lot about it, i.e. which use cases it supports, what it's designed to do, etc. But other people (or even you a year from now) may not have the same level of knowledge, even with documentation. So either new functions will pop up every time someone needs some slight modification in behavior (because they are not confident to change existing function), or someone may break whole bunch of acceptance tests.

   Best if those are true unit tests, able to run completely independently from anything (use mocks, predefined static test data, etc).

2. Protect yourself from unexpected changes / bugs in Selenium itself (or other important 3rd-party libraries). When you're updating Selenium to next version (and that usually needs to be done every 3-6 months), there's always a chance that they changed some default you were relying upon (and didn't even know), or broke something, or suddenly something returns a different exception, or does not throw exception where it previously did and so on. Of course no need to get carried away and duplicate Selenium own unit tests, but when it comes to non-trivial things, or relying on some features with poor documentation, those tests may help a lot.

   Those are integration tests. Ideally they should run against test-only webapp (not the real application) that replicate specifically tested behaviors in a way convenient for tests.

Of course some compromises possible as well. For example having a small subset of acceptance tests that are serving as unit / integration tests (they run first, and other tests only run if they are passing). Might be cheaper to begin with those and slowly migrate to proper unit/integration tests when you debug / fix issues in the test framework.

Another question is how do you separate tests testing your framework from actual acceptance tests for the product. What worked for me is keeping test framework and acceptance tests in 2 separate projects. That way I can change the framework, build it (which also includes running unit and integration tests) many times if needed. When all unit and integration tests are passing, then I can update the version used by actual acceptance tests.

Answer 2

Personally, I would not write test code that tests the test code.

Writing tests to shield myself from bugs in 3:rd party tools like Selenium is not something I would do. If the tests passes and the expected result is validated, then that would be enough for me.

When I upgrade to a new version of Selenium, I would do it when I had a working state. I.e. all tests passed. The only change would be the Selenium version. If I now have tests that breaks, I know that Selenium is behaving differently in this version than I expected and can act accordingly.

I would write tests for any complicated utility functionality I may need.

I would, however, work hard on writing the test code extremely easy to understand and avoid anything remotely complicated in it. And then be careful to verify that each test verified the behaviour I am expecting.