Run TestNG tests in random order

Question

Similarly to How can I make my JUnit tests run in random order? , I'd like TestNG to run my tests in random order, so unintended dependencies cannot creep in.

The TestNG manual states:

By default, TestNG will run the tests found in your testng.xml file in a random order.

However, I created a small test project, with a simple testng.xml:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="My suite">
    <test name="Simple test">
        <packages>
            <package name="testngtests"></package>
        </packages>
    </test>
</suite>

The package testngtests contains two test classes (MyTest1, MyTest2), and these contain a few empty methods like this:

@Test
public void testOne(){

}

The test mehods are all empty, and only differ by name.

When I run them (using the Eclipse TestNG runner, or on the command line), the tests consistenly run in the same order (namely sorted alphabetically, first by class and then by method name).

So is the documentation wrong?

Or does "in random order" simply mean "there is no guaranteed order"? Then how can I make TestNG actively randomize test order?

Answer 1

Yes, 'random' should really be 'non predictable'.

If you want true randomization, look up IMethodInterceptor, where TestNG will give you an opportunity to change the ordering to anything you like.

Answer 2

To expand on Cedric Beust's answer, here's my code for doing this, with some help from this example from GitHub:

import org.testng.IMethodInstance;
import org.testng.IMethodInterceptor;
import org.testng.ITestContext;
import java.util.Collections;
import java.util.List;
import java.util.Random;

public class TestOrderRandomizer implements IMethodInterceptor {
    @Override
    public List<IMethodInstance> intercept(List<IMethodInstance> methods, ITestContext context) {
        long seed = System.nanoTime();
        System.out.println(String.format("Randomizing order of tests with seed: %s", seed));
        Collections.shuffle(methods, new Random(seed));
        return methods;
    }
}

And to use it, add this before your test class:

import org.testng.annotations.Listeners;

@Listeners(TestOrderRandomizer.class)
public class TesterClass {
...

Printing the seed allows to reproduce a run-order by planting the same seed again.