Retry failed test case in testng

Question

I am working on a testng project and my goal is that let testng auto retry the failed test cases. For example, there are 10 test cases in the first round and 5 failed. So after first round, I let testng select the 5 failed test cases and rerun them again. In the second round, maybe there are 2 failed test cases, then I rerun this 2 agian.

I have tried the IRetryAnalyzer but it is different. The IRetryAnalyzer is retrying the failed test cases immediatelly instead of the end of each round.

So currently I want to call the retry using onStart and onFinish in the ISuiteListener. In this case I implement the onFinish method like this:

@Override
public void onFinish(ISuite suite) {
    logger.info("Round " + retryCounter
            + " Testing suit stops. onFinish method is invoked.");
    if (!doRetry()) {
        logger.info("Retry finished.");
        cleanReport(suite);
        return;
    }

    // I want to remove the passed cases here
    // and create a suite to run the failed test cases.
    suite.run();
}

So is it possible do that? Or any better idea for this requirement.

Remy Remy · Accepted Answer · 2016-03-30T22:22:57

It looks like the question is old, but still unanswered, so here is my solution.

Firstly, you have to implement IRetryAnalyzer interface like this:

    public class Retry implements IRetryAnalyzer {

    private int retryCount = 0;
    private int maxRetryCount = 1;

    public boolean retry(ITestResult result) {
        if (retryCount < maxRetryCount) {
            System.out.println("Retrying test " + result.getName() + " with status "
                    + getResultStatusName(result.getStatus()) + " for the " + (retryCount+1) + " time(s).");
            retryCount++;
            return true;
        }
        return false;
    }

    public String getResultStatusName(int status) {
        String resultName = null;
        if(status==1)
            resultName = "SUCCESS";
        if(status==2)
            resultName = "FAILURE";
        if(status==3)
            resultName = "SKIP";
        return resultName;
    }
}

Here, maxRetryCount is the number of times, you want your failed tests to be re-run. Tests are re-run immediately after their fail.

Secondly, implement IAnnotationTransformer interface:

public class RetryListener implements IAnnotationTransformer {

    @Override
    public void transform(ITestAnnotation testannotation, Class testClass,
                          Constructor testConstructor, Method testMethod)   {
        IRetryAnalyzer retry = testannotation.getRetryAnalyzer();

        if (retry == null)  {
            testannotation.setRetryAnalyzer(Retry.class);
        }

    }
}

Now, in your TestNG xml suite with tests, you should add listener:

<listeners>
    <listener class-name="Utils.reRunTest.RetryListener"/>
</listeners>

You can also add it to your pom.xml file:

 <properties>
     <property>
         <name>usedefaultlisteners</name>
         <value>false</value>
     </property>
     <property>
         <name>listener</name>
         <value>org.testng.reporters.EmailableReporter2, org.testng.reporters.XMLReporter, org.testng.reporters.FailedReporter, Utils.reRunTest.RetryListener
         </value>
     </property>

You may also want your test re-runs not to affect total number of tests. This is done, by implementing TestListener interface and adding it as a listener alongside with RetryAnalyzer. The implementation that will remove flaky tests and failed tests from total number of tests is:

public class TestListener implements ITestListener {
    @Override
    public void onFinish(ITestContext context) {
        Set<ITestResult> failedTests = context.getFailedTests().getAllResults();
        Set<ITestResult> skippedTests = context.getSkippedTests().getAllResults();
        for (ITestResult temp : failedTests) {
            ITestNGMethod method = temp.getMethod();
            if (context.getFailedTests().getResults(method).size() > 1) {
                failedTests.remove(temp);
            } else {
                if (context.getPassedTests().getResults(method).size() > 0) {
                    failedTests.remove(temp);
                }
            }
        }
        for (ITestResult temp : skippedTests) {
            ITestNGMethod method = temp.getMethod();
            if (context.getSkippedTests().getResults(method).size() > 1) {
                skippedTests.remove(temp);
            } else {
                if (context.getPassedTests().getResults(method).size() > 0) {
                    skippedTests.remove(temp);
                }
            }
        }
    }

    public void onTestStart(ITestResult result) {
    }

    public void onTestSuccess(ITestResult result) {
    }

    public void onTestFailure(ITestResult result) {
    }

    public void onTestSkipped(ITestResult result) {
    }

    public void onTestFailedButWithinSuccessPercentage(ITestResult result) {
    }

    public void onStart(ITestContext context) {
    }
}

Hope that helps. P.S Don't use it with TestNG version 6.9.10 `cause it has some issues and you re-run tests will always pass, disregarding their real state.

Retry failed test case in testng

2 Answers