QA testing tools integration

Question

I need some advice about choosing the right tools for my regression test strategy. The main criteria is "best buy" of the right tool. After analysis of Gartners case studies I'm trying to choose between Atlassian and IBM QA solution. The main dilemma is that IBM Jazz platform integrates Rational functional tester as the Automation testing solution, and Atlassian relies on Jira platform which doesn't integrate automation test toll by default.

My company is Java oriented so we eliminate HP Quick Test Professional because of VB script language.

Options are:

1. Atlassian: Jira - Greenhoper - Zephyr - Selenium 2
2. IBM: Jazz - Rational quality center - Rational team concert - Rational Functional tester.

The first one is obviously cheaper solution, but Selenium is in my opinion not long term solution cause it is harder to maintain huge number of test scripts when application is changing - RFT does map objects when I'm replacing something in scripts(or maybe I'm wrong).

Person who will be responsible for automation tests will be developer, cause most of the test I can't create only through the recorder.

Application to test is java web application and integration with another system.

Question: Is it possible that RFC integrates with Jira not with Jazz ? If yes, what do you think about Atlassian - Jira - Greenhoper - Zephyr - Rational Functional tester solution ?

Answer 1

I would not agree that Selenium tests are hard to manage, if they are written well they can be easy to maintain and extend. I agree though that they are bit harder to write, but in reality most of the more complex automated tests have to be written without recorder. Recorder is a tool to help writing tests faster, but not a tool to fully replace any coding.

If interface which is tested changes significantly, changing part of the tests is inevitable. In addition to Selenium you could use Jenkins which is a great tool for regression tests which will allow you to easily execute, schedule tests and integrate too. It provides also a variety of different plugins to use and its simple.

In longer run I would sugest to hire a tester with base coding skills to write the automated tests. The problem is, developers and qa have different mindset due the type of work they do. Developer is familiar with the code he writes, so the automated tests would relly on it, which can cause some bugs to be missed out ( in a way, that would be white box testing ) . On other side, QA are nit aware of the code, and they try to think as the users when writing the tests, which might not be as high quality as developer tests code wise, but in functionality testing they would be better ( in a way this would be black box testing ).

In summary I would definitely check out Selenium. Plus by writing the tests with code, you have an advantage to use programming language which is used for writing tests to extend them further. No recording tools could do that. Check out Jenkins too, it will allow tests to be ran by anyone on your team.

Answer 2

This is a complex question and not only because of what you are asking, you also need to take into account that if you start integrating between multiple vendors then your process becomes more fragile. This is not related to the quality of the integration or the vendors, it is only a matter of having multiple companies working separately that need to keep making changes that fit the changes made by the other companies... In short, this is another point against your first option that asks to integrate between solutions of 3 diff companies.

Having said that, even when you develop your testing framework with Selenium, you can do it in ways that will be scalable and that will support even when your objects are modified. You can use well defined functions and smart identification of objects. You should not make this your criteria for choosing either solution.

I guess that I would try both options (or any other options you have in the table), run a short pilot project with pros and cons for all of them and make your best decision. I think that it will be hard for you to decide otherwise (unless you are close to flipping a coin :-)

In my company we created a short whitepaper that may help you to structure your evaluation process. Here is the link, it is free BTW.

What also reminds me, Zephyr are not the only ones who integrate with Jira and know how to run Selenium (or TestComplete, or any other automation framework). You can also check out our solution. It's called PractiTest and it may be another option for you to try (sorry if instead of making things easier I added you another thing to consider :-)

In any case good luck!

-joel