Unexecuted test methods in TestNG 6.12+

Question

I've got the test classes with an XML suite as follows:

Class1:

public class TestReal1 {

    @Test
    public void testMethodClass1A_prio0() {
        // some testing code
    }

    @Test(priority = 10)
    public void testMethodClass1B_prio10() {
        // some testing code        
    }

    @Test(priority = 11)
    public void testMethodClass1C_prio11() {
        // some testing code
    }
}

Class2:

public class TestReal2 {

    @Test(priority = 1)
    public void testMethodClass2A_prio1() {
        // some testing code
    }

    @Test(priority = 2)
    public void testMethodClass2B_prio2() {
        // some testing code
    }

    @Test(priority = 3)
    public void testMethodClass2C_prio3() {
        // some testing code
    }
}

TestSuite.xml :

<?xml version="1.0" encoding="UTF-8"?>
<suite name="Suite" parallel="false">
    <test name="Test" preserve-order="true" verbose="10">
        <classes>
            <class name="com.test.radek.testngtest.TestReal1" />
            <class name="com.test.radek.testngtest.TestReal2" />
        </classes>
    </test>
</suite>

Having such configuration, for TestNG 6.12+ only two methods are executed. TestNG output:

===== Invoked methods
    TestReal1.testMethodClass1A_prio0()[pri:0, instance:com.test.radek.testngtest.TestReal1@54e156e1] 1424053985
    TestReal2.testMethodClass2A_prio1()[pri:1, instance:com.test.radek.testngtest.TestReal2@5a311ade] 1513167582
=====

In older versions all methods were executed. For example TestNG 6.10:

===== Invoked methods
    TestReal1.testMethodClass1A_prio0()[pri:0, instance:com.test.radek.testngtest.TestReal1@7adbe76f] 2061231983
    TestReal2.testMethodClass2A_prio1()[pri:1, instance:com.test.radek.testngtest.TestReal2@d029f4] 13642228
    TestReal2.testMethodClass2B_prio2()[pri:2, instance:com.test.radek.testngtest.TestReal2@d029f4] 13642228
    TestReal2.testMethodClass2C_prio3()[pri:3, instance:com.test.radek.testngtest.TestReal2@d029f4] 13642228
    TestReal1.testMethodClass1B_prio10()[pri:10, instance:com.test.radek.testngtest.TestReal1@7adbe76f] 2061231983
    TestReal1.testMethodClass1C_prio11()[pri:11, instance:com.test.radek.testngtest.TestReal1@7adbe76f] 2061231983
=====

I guess it has something to do with implementation of @Test attribute importance introduced in 6.10:

Hierarchy on order features (from less important to more important): groupByInstance, preserveOrder, priority, dependsOnGroups, dependsOnMethods

But the question is is it ok that in my case, using 6.12+ versions of TestNG some test cases are unexecuted? If yes, why? Is it something connected to changes in DynamicGraph in TestNG implementation?

What is the correct behaviuor now?

Krishnan Mahadevan Krishnan Mahadevan · Accepted Answer · 2018-06-06T03:29:42

The root cause of the problem is that in your suite xml file, your test classes are included in the wrong order.

TestReal1 has methods with lower priority (priorities 0, 10 and 11) and this class appears as the first one in the suite file.
TestReal2 has methods with higher priority(priorities 1, 2 and 3) and this class appears as the second one in the suite file.

This causes the confusion. AFAIK this issue has been fixed as a side effect of some fixes that were done to DynamicGraph and should no longer be a problem in TestNG 7.0.0-SNAPSHOT (soon to be released as TestNG 7.0.0)

Also please be advised that we are currently having an issue with TestNG 7.0.0-SNAPSHOT being published to Maven central.

So to fix this, you can try re-ordering your class in your suite xml (or) try using TestNG 7.0.0 series.

Unexecuted test methods in TestNG 6.12+

1 Answers