Remove (duplicate) failed TestNG result via test listener

Question

Similar to the solution posted here TestNG retrying failed tests doesn't output the correct test results, I'm trying to remove a (duplicate) test result using a test listener during onFinish(ITestContext context).

Although the removal of the result with context.getFailedTests().removeResult(result) appears to work fine (the result actually is removed), there seems to be "some other spot" where the results are being pulled from cause the build still fails.

Also note that when I run the sample test from the article above (which has one duplicate failure to be removed and one passed test), I get a difference in "test results" (not cleaned up as expected) vs. "suite results" (duplicate failure removed as expected).

And, where is the reporting pulling the results from to decide whether to fail the build? Or is it just that it's pulling the results before I'm cleaning up the failed tests ... ?

===============================================
    Default test
    Tests run: 3, Failures: 2, Skips: 0
===============================================

===============================================
Default suite
Total tests run: 2, Failures: 1, Skips: 0  
===============================================

EDIT: Just to clarify, we're running those tests with maven, and they are ITs, so we run them with the failsafe plugin. The problem is that even though it appears that the tests are removed, mvn verify still fails the build as it considers build failures to be found regardless.

And also if run such a test from Eclipse, even though the tests are removed, failures are still being printed in the log when the suite finishes.

About RetryAnalyzer: I would not consider using RetryAnalyzer good/best practice at all, but if you find yourself in a situation where you need to solve the problem, e.g. you inherited a test suite that is relying on RetryAnalyzer, you may find this useful.

We sort of figured out something"usable", although not what I ultimately wanted. I will try to find some time in the next few days to clean up the code and post it. — mac
Sorry, took way longer than I promised, but our solution is below. If you have any questions, please post them below. — mac
Ha! Hold on! I just realized that the other solution in the other TestNG thread is actually ... yours! Well ... anyhow, the main thing that I changed is that we're running mvn integration-test and trust the TestNG plugin to fail the run if necessary based on the TestNG result file. Clean up will be useless if you do try it with mvn verify — mac

Morvader Morvader · Accepted Answer · 2015-03-18T14:55:37

Try using this code:

ListenerApadter:

public class MyTestListenerAdapter extends TestListenerAdapter {
    @Override
    public void onTestFailure(ITestResult result) {
        if (result.getMethod().getRetryAnalyzer() != null) {
            MyRetryAnalyzer retryAnalyzer = (MyRetryAnalyzer)result.getMethod().getRetryAnalyzer();

            if(retryAnalyzer.isRetryAvailable()) {
                result.setStatus(ITestResult.SKIP);
            } else {
                result.setStatus(ITestResult.FAILURE);
            }
            Reporter.setCurrentTestResult(result);
        }
    }

   @Overrride
   public void onFinish(ITestContext context) {
     Iterator<ITestResult> failedTestCases =context.getFailedTests().getAllResults().iterator();
    while (failedTestCases.hasNext()) {
        System.out.println("failedTestCases");
        ITestResult failedTestCase = failedTestCases.next();
        ITestNGMethod method = failedTestCase.getMethod();
        if (context.getFailedTests().getResults(method).size() > 1) {
            System.out.println("failed test case remove as dup:" + failedTestCase.getTestClass().toString());
            failedTestCases.remove();
        } else {

            if (context.getPassedTests().getResults(method).size() > 0) {
                System.out.println("failed test case remove as pass retry:" + failedTestCase.getTestClass().toString());
                failedTestCases.remove();
            }
        }
    }
   }
}

RetryAnalizer:

public class MyRetryAnalyzer implements IRetryAnalyzer {
    private static int MAX_RETRY_COUNT = 3;

    AtomicInteger count = new AtomicInteger(MAX_RETRY_COUNT);

    public boolean isRetryAvailable() {
        return (count.intValue() > 0);
    }

    @Override
    public boolean retry(ITestResult result) {
        boolean retry = false;
        if (isRetryAvailable()) {
            System.out.println("Going to retry test case: " + result.getMethod() + ", " + (MAX_RETRY_COUNT - count.intValue() + 1) + " out of " + MAX_RETRY_COUNT);
            retry = true;
            count.decrementAndGet();
        }
        return retry;
    }
}

POM.xml -> Surefire Configuration:

This is where you should configure "overwrite" surefire listener wich has their own counters.

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.18.1</version>
  <configuration>
    <suiteXmlFiles><suiteXmlFile>${basedir}/testng.xml</suiteXmlFile></suiteXmlFiles>
 <properties> 
   <property>
    <name>listener</name>
    <value>Utils.MyTestListenerAdapter,Utils.MyRetryAnalizer</value>
   </property>
 </properties>

Remove (duplicate) failed TestNG result via test listener

3 Answers