Issues in order of execution of tests in TestNG

Question

I have a package- 'scripts'

Under scripts, I have two classes- A and B

Under A class I have 3 methods (@Test) written in order-

@Test(priority=1)
public void b(){ }

@Test(priority=2)
public void a(){}

@Test(priority=3)
public void d(){}

Under B class I have 1 method-

@Test
public void c(){}

TestNG.XML

<classes>
  <class name="scripts.A"/>
  <class name="scripts.B"/>
</classes>

On Execution- as a TestNG Suite- Right click on TestNG XML and select-Run as a TestNG Suite

Class B executes first- This is because @Test in class B has no priority specified and thus, becomes of highest priority

Is there any way by which priorities of @Test should matter only within the class?Why TestNG compares priorities of all the test methods from all the classes and then plan execution. Moreover, since I am defining the execution order of the classes in TestNG XML, it's obvious that I want the second class methods to be executed after first class methods. In this case, expected execution order should- first A's methods and then B's methods.

When I remove priorities from each @Test and the starts execution, then executions doesn't occur in an order in which the methods are written. In fact, execution starts in an ascending order of the method names. In this case, order is- a(),b(),c(),d()

How to get away with that?

I am also wondering about its order of execution. But if you want to execute suite in your order move scripts.B to top and scripts.A to down. Now all scripts will execute in order. — Shiva krishna Chippa
Even like that, it doesn't work. TestNG still takes priority of @test methods first and executes accordingly. — Dhruv Bhatnagar

Krishnan Mahadevan Krishnan Mahadevan · Accepted Answer · 2017-10-16T05:46:53

TestNG by default first gathers all methods and then sorts them based on their priorities (It doesn't classify them on the class in which they occur before). That being said, here's how you do it.

First ensure that your @Test methods don't use priority attribute (or) you have explicitly set preserve-order as false in your suite xml file.
You then build a method interceptor which first orders methods based on the order of the classes as they occur in the suite xml file, and then using a custom annotation which kind of serves the same priority ordering mechanism.

That should take care of your needs.

Here's your A class look-alike:

import org.testng.ITestResult;
import org.testng.Reporter;
import org.testng.annotations.Test;

public class TestClassASample {
    @Test
    @PriorityInClass(priority = 1)
    public void a() {
        System.err.println(stringify());
    }

    @Test
    @PriorityInClass(priority = 2)
    public void b() {
        System.err.println(stringify());
    }

    @Test
    @PriorityInClass(priority = 3)
    public void c() {
        System.err.println(stringify());
    }

    private String stringify() {
        ITestResult result = Reporter.getCurrentTestResult();
        return result.getTestClass().getRealClass().getName() + "." + result.getMethod().getMethodName() + "()";
    }
}

Here's your B class look-alike :

import org.testng.ITestResult;
import org.testng.Reporter;
import org.testng.annotations.Test;

public class TestClassBSample {
    @Test
    public void c() {
        System.err.println(stringify());
    }

    private String stringify() {
        ITestResult result = Reporter.getCurrentTestResult();
        return result.getTestClass().getRealClass().getName() + "." + result.getMethod().getMethodName() + "()";
    }
}

Here's how your custom annotation looks like :

import java.lang.annotation.Retention;
import java.lang.annotation.Target;

import static java.lang.annotation.ElementType.METHOD;

@Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
@Target({METHOD})
public @interface PriorityInClass {
    int priority() default 0;
}

Here's how the method interceptor would look like :

import org.testng.IMethodInstance;
import org.testng.IMethodInterceptor;
import org.testng.ITestContext;
import org.testng.xml.XmlClass;

import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;

public class MethodOrderer implements IMethodInterceptor {
    @Override
    public List<IMethodInstance> intercept(List<IMethodInstance> methods, ITestContext context) {
        Map<Class, List<IMethodInstance>> mapping = new LinkedHashMap<>();
        List<XmlClass> xmlClasses = context.getCurrentXmlTest().getXmlClasses();
        for (XmlClass xmlClass : xmlClasses) {
            Class<?> clazz = xmlClass.getSupportClass();
            mapping.put(clazz, new ArrayList<>());
            for (IMethodInstance method : methods) {
                Class<?> methodClass = method.getMethod().getTestClass().getRealClass();
                if (mapping.containsKey(methodClass)) {
                    List<IMethodInstance> methodInstances = mapping.get(methodClass);
                    if (!methodInstances.contains(method)) {
                        methodInstances.add(method);
                    }
                }
            }
        }
        List<IMethodInstance> returnValue = new ArrayList<>(methods.size());
        Sorter sorter = new Sorter();
        for (Map.Entry<Class, List<IMethodInstance>> each : mapping.entrySet()) {
            List<IMethodInstance> methodInstances = each.getValue();
            Collections.sort(methodInstances, sorter);
            returnValue.addAll(methodInstances);
        }
        return returnValue;
    }

    public static class Sorter implements Comparator<IMethodInstance> {

        @Override
        public int compare(IMethodInstance o1, IMethodInstance o2) {
            PriorityInClass o1Priority = getPriority(o1);
            PriorityInClass o2Priority = getPriority(o2);
            int priorityO1 = (o1Priority == null ? getPriorityViaTestAnnotation(o1) : o1Priority.priority());
            int priorityO2 = (o2Priority == null ? getPriorityViaTestAnnotation(o1) : o2Priority.priority());
            return (priorityO1 < priorityO2) ? -1 : ((priorityO1 == priorityO2) ? 0 : 1);
        }

        private PriorityInClass getPriority(IMethodInstance instance) {
            return instance.getMethod().getConstructorOrMethod().getMethod().getAnnotation(PriorityInClass.class);
        }

        private int getPriorityViaTestAnnotation(IMethodInstance instance) {
            return instance.getMethod().getPriority();
        }
    }
}

Here's how the suite xml would look like

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="suite">
    <listeners>
        <listener class-name="com.rationaleemotions.stackoverflow.qn46748930.MethodOrderer"/>
    </listeners>
    <test name="random_test"  preserve-order="false">
        <classes>
            <class name="com.rationaleemotions.stackoverflow.qn46748930.TestClassASample"/>
            <class name="com.rationaleemotions.stackoverflow.qn46748930.TestClassBSample"/>
        </classes>
    </test> <!-- random_test -->
</suite> <!-- suite -->

Here's the output :

com.rationaleemotions.stackoverflow.qn46748930.TestClassASample.a()
com.rationaleemotions.stackoverflow.qn46748930.TestClassASample.b()
com.rationaleemotions.stackoverflow.qn46748930.TestClassASample.c()
com.rationaleemotions.stackoverflow.qn46748930.TestClassBSample.c()

===============================================
suite
Total tests run: 4, Failures: 0, Skips: 0
===============================================

Issues in order of execution of tests in TestNG

1 Answers