How to order TestNG Factory execution given a single method test and sortable test data?

Question

First of all you should know that I am using TestNG v6.8.8 and Java JDK 6. I experience this problem running on different flavors of Linux and Mac OS 10.9.4. On to the code listing.

Factory class

import org.testng.annotations.Factory;

import java.util.ArrayList;
import java.util.Collections;
import java.util.List;

/**
 * @author rcourtright
 *         Date: 8/5/14
 *         Time: 4:50 PM
 */
public class ErsatzFactory {

    private final List<String> testData;

    public ErsatzFactory() {
        testData = new ArrayList<String>();
        int order = 0;
        for (int i = 0 ; i < 9; i++) {
            testData.add(order++ + "-Test");
        }
        Collections.sort(testData);
    }

    @Factory
    public Object[] setUp() {
        List<ErsatzTest> objects = new ArrayList<ErsatzTest>();
        int order = 0;
        for (String testDatum : testData) {
            objects.add(
                    order,
                    new ErsatzTest(testDatum)
            );
            order++;
        }
        return objects.toArray();
    }

}

Test class

import org.testng.ITest;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

/**
 * @author rcourtright
 *         Date: 8/5/14
 *         Time: 4:45 PM
 */
//@Listeners(ErsatzListener.class)
public class ErsatzTest implements ITest {

    private String order;

    public ErsatzTest(String order) {
        this.order = order;
    }

    @Test
    public void justDoIt() {
        System.out.println(order);
    }

    @Override
    public String getTestName() {
        return order;
    }
}

TestNG XML

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

<suite name="ersatz-testing" verbose="1" >
    <test name="ersatz-test-factory" preserve-order="true">
        <classes>
            <class name="ErsatzFactory"/>
        </classes>
    </test>
</suite>

So you can see in my construction I have a list which has a natural order and I sort that collection. It is the job of the Factory to execute the single test method in that order. By debugging I have found that the collection returned by the Factory is in the desired order. But that is not how they are executed.

Using TestNG supported ant target, this is the order of execution:

Test results

ersatz-test-executor:
   [testng] 5-Test
   [testng] 2-Test
   [testng] 7-Test
   [testng] 6-Test
   [testng] 4-Test
   [testng] 8-Test
   [testng] 1-Test
   [testng] 0-Test
   [testng] 3-Test
   [testng]
   [testng] ===============================================
   [testng] ersatz-testing
   [testng] Total tests run: 9, Failures: 0, Skips: 0
   [testng] ===============================================
   [testng]

This is both unexpected and undesirable. Clearly this test is trivial, but it is a proxy for a rather complex system test for which I must get the order of execution correct.

Although not listed here, I must use a TestNG Listener for post test result processing and it appears that I must use a Factory as opposed to a stand-alone DataProvider. (I have commented out the Listener annotation in the test class declaration.) I should note that the Listener works without incident.

Because there is one and only one method, I cannot use the Priority annotation or method dependencies. The test is completely data driven. The data is sortable and, if memory serves, when I used a DataProvider from within the test class I got that order. But I also need the Listener for decorating the test results reports and that has driven me to using the factory. I should note that the execution problem exists whether I use the listener or not. If you compile and run this code I expect you will see the results listed in what appears to be a random order.

Thanks in advance for your consideration of this problem.

Answer 1

Okay, I sussed it. I did not appreciate the pre-processing power of a method interceptor. So I created one and I got my ordered tests. For the benefit of anyone seeking a solution I will list my code changes here.

Method interceptor

import org.testng.IMethodInstance;
import org.testng.IMethodInterceptor;
import org.testng.ITestContext;

import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;

/**
 * @author rcourtright
 *         Date: 8/11/14
 *         Time: 2:37 PM
 */
public class ErsatzMethodInterceptor implements IMethodInterceptor {

    @Override
    public List<IMethodInstance> intercept(List<IMethodInstance> methods, ITestContext context) {
        Map<String,IMethodInstance> orders = new TreeMap<String,IMethodInstance>();
        for (IMethodInstance iMethodInstance : methods) {
            if (!iMethodInstance.getMethod().getMethodName().equals("justDoIt")) {
                continue;
            }
            Object obj = iMethodInstance.getInstance();
            if (!(obj instanceof ErsatzTest)) {
                continue;
            }
            ErsatzTest ersatzTest = (ErsatzTest)obj;
            String order = ersatzTest.getOrder();
            orders.put(order,iMethodInstance);
        }
        List<IMethodInstance> tests = new ArrayList<IMethodInstance>(orders.size());
        for (String order : orders.keySet()) {
            IMethodInstance test = orders.get(order);
            tests.add(test);
        }
        return tests;
    }
}

Wire the test class to the interceptor class

    import org.apache.log4j.Logger;
import org.testng.ITest;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

/**
 * @author rcourtright
 *         Date: 8/5/14
 *         Time: 4:45 PM
 */
@Listeners(
    {
        ErsatzMethodInterceptor.class,
        ErsatzListener.class
    }
)
public class ErsatzTest implements ITest {
    private Logger logger = Logger.getLogger(ErsatzTest.class);

    private String order;

    public ErsatzTest(String order) {
        this.order = order;
    }

    public String getOrder() {
        return order;
    }

    @Test
    public void justDoIt() {
        logger.info(order);
    }

    @Override
    public String getTestName() {
        return order;
    }
}

Bonus listener to pretty print results to reports

    import org.apache.log4j.Logger;
import org.testng.ITestResult;
import org.testng.TestListenerAdapter;
import org.testng.internal.BaseTestMethod;
import org.testng.internal.TestResult;

import java.lang.reflect.Field;

/**
 * @author rcourtright
 *         Date: 7/29/14
 *         Time: 12:03 PM
 */
public class ErsatzListener extends TestListenerAdapter {

    private Logger logger = Logger.getLogger(ErsatzListener.class);

    private void setTestNameInXml(ITestResult tr) {
        try {
            Field mMethod = TestResult.class.getDeclaredField("m_method");
            mMethod.setAccessible(true);
            mMethod.set(tr, tr.getMethod().clone());
            Field mMethodName = BaseTestMethod.class.getDeclaredField("m_methodName");
            mMethodName.setAccessible(true);
            mMethodName.set(tr.getMethod(), tr.getTestName());
        } catch (IllegalAccessException ex) {
            logger.error(ex.getLocalizedMessage(), ex);
        } catch (NoSuchFieldException ex) {
            logger.error(ex.getLocalizedMessage(), ex);
        }
    }

    @Override
    public void onTestSuccess(ITestResult tr) {
        setTestNameInXml(tr);
        super.onTestSuccess(tr);
    }

    @Override
    public void onTestFailure(ITestResult tr) {
        setTestNameInXml(tr);
        super.onTestFailure(tr);
    }

    @Override
    public void onTestSkipped(ITestResult tr) {
        setTestNameInXml(tr);
        super.onTestSkipped(tr);
    }

}

I assume that you have log4j now, so just have a usable log4j.xml on the class path and here we have ordered test results:

$ java org.testng.TestNG testng.xml
[TestNG] Running:
  /Users/rcourtright/Desktop/ersatz/testng.xml


0 [main] INFO ErsatzTest  - 0-Test

2 [main] INFO ErsatzTest  - 1-Test

3 [main] INFO ErsatzTest  - 2-Test

4 [main] INFO ErsatzTest  - 3-Test

5 [main] INFO ErsatzTest  - 4-Test

6 [main] INFO ErsatzTest  - 5-Test

7 [main] INFO ErsatzTest  - 6-Test

8 [main] INFO ErsatzTest  - 7-Test

9 [main] INFO ErsatzTest  - 8-Test

===============================================
ersatz-testing
Total tests run: 9, Failures: 0, Skips: 0
===============================================

In the end, I did not appreciate the power of the method interceptor and that it runs after the factory does whatever it does. That is, the factory method does not control the order, but this can be changed by using an interceptor.