Test automation with Jenkins + Robot framework

0
votes

We are using Jenkins for the Continuous Integration.

I have also created a Jenkins job for robot framework test cases. But am facing an issue while running robot framework test cases using Jenkins.

Say, if I have to include around 300 test cases in Jenkins job for robot framework, while running the test cases, if any of the test case fails, then the remaining test scripts will not be get executed.

For example : while running the test cases , if 10th test case fails, then the remaining 290 test cases will not be get executed !

Is there any better option in Jenkins to run all test cases in a concurrent manner ? - So that failure of any test cases will not effect the execution of other test cases !

My only requirement is - All the test cases listed in Jenkins job must be get executed even if any test cases got failed in between.

Can you please suggest a solution for this issue ?

Do I have to include any other plugins for this ?

Here is the screenshot of this particular Jenkins Job - enter image description here

2
jenkinsselenium-webdriverautomationautomated-testsrobotframework
how are you running all of these tests? What does your jenkins job look like? This doesn't sound like normal behavior. And by "any one test case fails" do you literally mean a single test case, or do you have just one test case but 300 keywords that do assertions? - Bryan Oakley
Hello @Bryan ..Thank you for your response. I am relatively new to Jenkins. I have edited my question & added a screenshot of this particular Jenkins job. Can you please go through it ? I am running each test cases by executing a shell which contains command pybot /path/to/test/case .I just created a single job which contains 300 shells ( each executing a command pybot ........ ) . Is it the right way to create a Jenkins job for this purpose ? & "any one test case fails" means if any of the test case fails ( pybot .... ), the remaining shells will not be get executed. - Arun Ramachandran

2 Answers

3
votes

That's not a good way to be running your tests in a batch. For starters, the RobotFramework log would only contain the results of the last test, even if they all passed.

The issue is, when an 'Execute Shell' step in the build fails, Jenkins stops the build. As pointed out in the comments above, you could try changing the return state with exit 0.

However, instead of each test being a separate build step with:

pybot /home/arun/test-cases/228.txt
pybot /home/arun/test-cases/117.txt
...etc

You should be executing them together in one step:

pybot /home/arun/test-cases/

If you need to control which tests execute and not all the valid test files in that folder and its subfolders, then use tags.

1
votes

https://wiki.jenkins-ci.org/display/JENKINS/Robot+Framework+Plugin lists as one of the necessary configuration steps:

  1. Force your Robot script to return successfully from shell with "exit 0" to empower the plugin in deciding if the build is success/failure (by default Robot exits with error code when there's any failed tests)

Thus if you have your test scripts return 0, then the jenkins build won't fail and you can decide upon the outcome what status you want to display.

  1. Set thresholds and optionally disable thresholds for critical tests only to count every test in the pass percentage.