Controlling execution order of unit tests in Visual Studio

Question

Okay, I'm done searching for good information on this. I have a series of Unit Tests that call a static class which, once initialized, sets properties that cannot (or I don't wish to) change.

My problem is I cannot enforce a set order for the tests to run. If I could, I could run them in such a way as the static properties would be set in a reliable way, and I could Assert on them, but unfortunately the Microsoft.VisualStudio.TestTools.UnitTesting framework just runs them in a seemingly random order.

So, I found this http://msdn.microsoft.com/en-us/library/microsoft.visualstudio.testtools.unittesting.priorityattribute.aspx which says in the Remarks section "This attribute is not used by the test system. It is provided to the user for custom purposes." Huh? What good is it then? Do they expect me to write my own testing wrapper to take advantage of this fabulous attribute (of which I could easily write myself if I wanted to go to that level of effort...)

So, enough of the rant; Bottom line, is there a way to control the order my unit tests run?

[TestMethod]
[Priority(0)]

etc. does NOT seem to work, which makes sense, since Microsoft says it won't.

Also, please no comments about "violating isolation". The TestClass isolates what I am testing, not the individual TestMethods. Regardless, each test can be run independently just fine, they just can't be run together in a random order as there is no way to tear down the static class.

Oh, I also know about "Ordered Test".

Answer 1

Merge your tests into one giant test will work. To make the test method more readable, you can do something like

[TestMethod]
public void MyIntegratonTestLikeUnitTest()
{
    AssertScenarioA();

    AssertScenarioB();

    ....
}

private void AssertScenarioA()
{
     // Assert
}

private void AssertScenarioB()
{
     // Assert
}

Actually the issue you have suggests you probably should improve the testability of the implementation.

Answer 2

You can Use Playlist

Right click on the test method -> Add to playlist -> New playlist

the execution order will be as you add them to the playlist but if you want to change it you have the file

Answer 3

As you should know by now, purists say it's forbiden to run ordered tests. That might be true for unit tests. MSTest and other Unit Test frameworks are used to run pure unit test but also UI tests, full integration tests, you name it. Maybe we shouldn't call them Unit Test frameworks, or maybe we should and use them according to our needs. That's what most people do anyway.

I'm running VS2015 and I MUST run tests in a given order because I'm running UI tests (Selenium).

Priority - Doesn't do anything at all This attribute is not used by the test system. It is provided to the user for custom purposes.

orderedtest - it works but I don't recommend it because:

1. An orderedtest a text file that lists your tests in the order they should be executed. If you change a method name, you must fix the file.
2. The test execution order is respected inside a class. You can't order which class executes its tests first.
3. An orderedtest file is bound to a configuration, either Debug or Release
4. You can have several orderedtest files but a given method can not be repeated in different orderedtest files. So you can't have one orderedtest file for Debug and another for Release.

Other suggestions in this thread are interesting but you loose the ability to follow the test progress on Test Explorer.

You are left with the solution that purist will advise against, but in fact is the solution that works: sort by declaration order.

The MSTest executor uses an interop that manages to get the declaration order and this trick will work until Microsoft changes the test executor code.

This means the test method that is declared in the first place executes before the one that is declared in second place, etc.

To make your life easier, the declaration order should match the alphabetical order that is is shown in the Test Explorer.

• A010_FirstTest
• A020_SecondTest
• etc
• A100_TenthTest

I strongly suggest some old and tested rules:

• use a step of 10 because you will need to insert a test method later on
• avoid the need to renumber your tests by using a generous step between test numbers
• use 3 digits to number your tests if you are running more than 10 tests
• use 4 digits to number your tests if you are running more than 100 tests

VERY IMPORTANT

In order to execute the tests by the declaration order, you must use Run All in the Test Explorer.

Say you have 3 test classes (in my case tests for Chrome, Firefox and Edge). If you select a given class and right click Run Selected Tests it usually starts by executing the method declared in the last place.

Again, as I said before, declared order and listed order should match or else you'll in big trouble in no time.

Answer 4

I dont see anyone mentioning the ClassInitialize attribute method. The attributes are pretty straight forward.

Create methods that are marked with either the [ClassInitialize()] or [TestInitialize()] attribute to prepare aspects of the environment in which your unit test will run. The purpose of this is to establish a known state for running your unit test. For example, you may use the [ClassInitialize()] or the [TestInitialize()] method to copy, alter, or create certain data files that your test will use.

Create methods that are marked with either the [ClassCleanup()] or [TestCleanUp{}] attribute to return the environment to a known state after a test has run. This might mean the deletion of files in folders or the return of a database to a known state. An example of this is to reset an inventory database to an initial state after testing a method that is used in an order-entry application.

• [ClassInitialize()] Use ClassInitialize to run code before you run the first test in the class.

• [ClassCleanUp()] Use ClassCleanup to run code after all tests in a class have run.

• [TestInitialize()] Use TestInitialize to run code before you run each test.

• [TestCleanUp()] Use TestCleanup to run code after each test has run.

Answer 5

Since you've already mentioned the Ordered Test functionality that the Visual Studio testing framework supplies, I'll ignore that. You also seem to be aware that what you're trying to accomplish in order to test this Static Class is a "bad idea", so I'll ignore that to.

Instead, lets focus on how you might actually be able to guarantee that your tests are executed in the order you want. One option (as supplied by @gaog) is "one test method, many test functions", calling your test functions in the order that you want from within a single function marked with the TestMethod attribute. This is the simplest way, and the only disadvantage is that the first test function to fail will prevent any of the remaining test functions from executing.

With your description of the situation, this is the solution I would suggest you use.

If the bolded part is a problem for you, you can accomplish an ordered execution of isolated tests by leveraging the in built data driven test functionality. Its more complicated and feels a bit dirty, but it gets the job done.

In short, you define a data source (like a CSV file, or a database table) that controls the order in which you need to run your tests, and names of the functions that actually contain the test functionality. You then hook that data source into a data driven test, use the sequential read option, and execute your functions, in the order you want, as individual tests.

[TestClass]
public class OrderedTests
{
    public TestContext TestContext { get; set; }

    private const string _OrderedTestFilename = "TestList.csv";

    [TestMethod]
    [DeploymentItem(_OrderedTestFilename)]
    [DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV", _OrderedTestFilename, _OrderedTestFilename, DataAccessMethod.Sequential)]
    public void OrderedTests()
    {
        var methodName = (string)TestContext.DataRow[0];
        var method = GetType().GetMethod(methodName);
        method.Invoke(this, new object[] { });
    }

    public void Method_01()
    {
        Assert.IsTrue(true);
    }

    public void Method_02()
    {
        Assert.IsTrue(false);
    }

    public void Method_03()
    {
        Assert.IsTrue(true);
    }
}

In my example, I have a supporting file called TestList.csv, which gets copied to output. It looks like this:

TestName
Method_01
Method_02
Method_03

Your tests will be executed in the order that you specified, and in normal test isolation (i.e. if one fails, the rest still get executed, but sharing static classes).

The above is really only the basic idea, if I were to use it in production I would generate the test function names and their order dynamically before the test is run. Perhaps by leveraging PriorityAttribute you found and some simple reflection code to extract the test methods in the class and order them appropriately, then write that order to the data source.

Answer 6

Here is a class that can be used to setup and run ordered tests independent of MS Ordered Tests framework for whatever reason--like not have to adjust mstest.exe arguments on a build machine, or mixing ordered with non-ordered in a class.

The original testing framework only sees the list of ordered tests as a single test so any init/cleanup like [TestInitalize()] Init() is only called before and after the entire set.

Usage:

        [TestMethod] // place only on the list--not the individuals
        public void OrderedStepsTest()
        {
            OrderedTest.Run(TestContext, new List<OrderedTest>
            {
                new OrderedTest ( T10_Reset_Database, false ),
                new OrderedTest ( T20_LoginUser1, false ),
                new OrderedTest ( T30_DoLoginUser1Task1, true ), // continue on failure
                new OrderedTest ( T40_DoLoginUser1Task2, true ), // continue on failure
                // ...
            });                
        }

Implementation:

using Microsoft.VisualStudio.TestTools.UnitTesting;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;

namespace UnitTests.Utility
{    
    /// <summary>
    /// Define and Run a list of ordered tests. 
    /// 2016/08/25: Posted to SO by crokusek 
    /// </summary>    
    public class OrderedTest
    {
        /// <summary>Test Method to run</summary>
        public Action TestMethod { get; private set; }

        /// <summary>Flag indicating whether testing should continue with the next test if the current one fails</summary>
        public bool ContinueOnFailure { get; private set; }

        /// <summary>Any Exception thrown by the test</summary>
        public Exception ExceptionResult;

        /// <summary>
        /// Constructor
        /// </summary>
        /// <param name="testMethod"></param>
        /// <param name="continueOnFailure">True to continue with the next test if this test fails</param>
        public OrderedTest(Action testMethod, bool continueOnFailure = false)
        {
            TestMethod = testMethod;
            ContinueOnFailure = continueOnFailure;
        }

        /// <summary>
        /// Run the test saving any exception within ExceptionResult
        /// Throw to the caller only if ContinueOnFailure == false
        /// </summary>
        /// <param name="testContextOpt"></param>
        public void Run()
        {
            try
            {
                TestMethod();
            }
            catch (Exception ex)
            {
                ExceptionResult = ex;
                throw;
            }
        }

        /// <summary>
        /// Run a list of OrderedTest's
        /// </summary>
        static public void Run(TestContext testContext, List<OrderedTest> tests)
        {
            Stopwatch overallStopWatch = new Stopwatch();
            overallStopWatch.Start();

            List<Exception> exceptions = new List<Exception>();

            int testsAttempted = 0;
            for (int i = 0; i < tests.Count; i++)
            {
                OrderedTest test = tests[i];

                Stopwatch stopWatch = new Stopwatch();
                stopWatch.Start();

                testContext.WriteLine("Starting ordered test step ({0} of {1}) '{2}' at {3}...\n",
                    i + 1,
                    tests.Count,
                    test.TestMethod.Method,
                    DateTime.Now.ToString("G"));

                try
                {
                    testsAttempted++;
                    test.Run();
                }
                catch
                {
                    if (!test.ContinueOnFailure)
                        break;
                }
                finally
                {
                    Exception testEx = test.ExceptionResult;

                    if (testEx != null)  // capture any "continue on fail" exception
                        exceptions.Add(testEx);

                    testContext.WriteLine("\n{0} ordered test step {1} of {2} '{3}' in {4} at {5}{6}\n",
                        testEx != null ? "Error:  Failed" : "Successfully completed",
                        i + 1,
                        tests.Count,
                        test.TestMethod.Method,
                        stopWatch.ElapsedMilliseconds > 1000
                            ? (stopWatch.ElapsedMilliseconds * .001) + "s"
                            : stopWatch.ElapsedMilliseconds + "ms",
                        DateTime.Now.ToString("G"),
                        testEx != null
                            ? "\nException:  " + testEx.Message +
                                "\nStackTrace:  " + testEx.StackTrace +
                                "\nContinueOnFailure:  " + test.ContinueOnFailure
                            : "");
                }
            }

            testContext.WriteLine("Completed running {0} of {1} ordered tests with a total of {2} error(s) at {3} in {4}",
                testsAttempted,
                tests.Count,
                exceptions.Count,
                DateTime.Now.ToString("G"),
                overallStopWatch.ElapsedMilliseconds > 1000
                    ? (overallStopWatch.ElapsedMilliseconds * .001) + "s"
                    : overallStopWatch.ElapsedMilliseconds + "ms");

            if (exceptions.Any())
            {
                // Test Explorer prints better msgs with this hierarchy rather than using 1 AggregateException().
                throw new Exception(String.Join("; ", exceptions.Select(e => e.Message), new AggregateException(exceptions)));
            }
        }
    }
}

Answer 7

I'll not address the order of tests, sorry. Others already did it. Also, if you know about "ordered tests" - well, this is MS VS's response to the problem. I know that those ordered-tests are no fun. But they thought it will be "it" and there's really nothing more in MSTest about that.

I write about one of your assumptions:

as there is no way to tear down the static class.

Unless your static class represents some process-wide external state external to your code (like ie. the state of an unmanaged native DLL library thats P/Invoked by the rest of your code), your assumption that there is no way is not true.

If your static class refers to this, then sorry, you are perfectly right, the rest of this anwer is irrelevant. Still, as you didn't say that, I assume your code is "managed".

Think and check the AppDomain thingy. Rarely it is needed, but this is exactly the case when you'd probably like to use them.

You can create a new AppDomain, and instantiate the test there, and run the test method there. Static data used by managed code will isolated there and upon completion, you will be able to unload the AppDomain and all the data, statics included, will evaporate. Then, next test would initialize another appdomain, and so on.

This will work unless you have external state that you must track. AppDomains only isolate the managed memory. Any native DLL will still be load per-process and their state will be shared by all AppDomains.

Also, creating/tearing down the appdomains will, well, slow down the tests. Also, you may have problems with assembly resolution in the child appdomain, but they are solvable with reasonable amount of reusable code.

Also, you may have small problems with passing test data to - and back from - the child AppDomain. Objects passed will either have to be serializable in some way, or be MarshalByRef or etc. Talking cross-domain is almost like IPC.

However, take care here, it will be 100% managed talking. If you take some extra care and add a little work to the AppDomain setup, you will be able to even pass delegates and run them in the target domain. Then, instead of making some hairy cross-domain setup, you can wrap your tests with to something like:

void testmethod()
{
    TestAppDomainHelper.Run( () =>
    {
        // your test code
    });
}

or even

[IsolatedAppDomain]
void testmethod()
{
    // your test code
}

if your test framework supports creating such wrappers/extensions. After some initial research and work, using them is almost trivial.

Answer 8

I see that this topic is almost 6 years old, and we now have new version of Visual studio but I will reply anyway. I had that order problem in Visual Studio 19 and I figured it out by adding capital letter (you can also add small letter) in front of your method name and in alphabetical order like this:

[TestMethod]
        public void AName1()
        {}
[TestMethod]
        public void BName2()
        {}

And so on. I know that this doesn't look appealing, but it looks like Visual is sorting your tests in test explorer in alphabetical order, doesn't matter how you write it in your code. Playlist didn't work for me in this case.

Hope that this will help.

Answer 9

If you can use the NUnit framwork, it is possible using the [Order] attribute.

see the MS doc for tests ordering using NUnit:

using NUnit.Framework;

namespace NUnit.Project
{
    public class ByOrder
    {
        public static bool Test1Called;
        public static bool Test2ACalled;
        public static bool Test2BCalled;
        public static bool Test3Called;

        [Test, Order(5)]
        public void Test1()
        {
            Test3Called = true;

            Assert.IsTrue(Test1Called);
            Assert.IsFalse(Test2ACalled);
            Assert.IsTrue(Test2BCalled);
        }

        [Test, Order(0)]
        public void Test2B()
        {
            Test2BCalled = true;

            Assert.IsTrue(Test1Called);
            Assert.IsFalse(Test2ACalled);
            Assert.IsFalse(Test3Called);
        }

        [Test]
        public void Test2A()
        {
            Test2ACalled = true;

            Assert.IsTrue(Test1Called);
            Assert.IsTrue(Test2BCalled);
            Assert.IsTrue(Test3Called);
        }

        [Test, Order(-5)]
        public void Test3()
        {
            Test1Called = true;

            Assert.IsFalse(Test2ACalled);
            Assert.IsFalse(Test2BCalled);
            Assert.IsFalse(Test3Called);
        }
    }
} 

Answer 10

they just can't be run together in a random order as there is no way to tear down the static class

You can name namespaces and classes in alphabetical order. eg.:

• MyApp.Test.Stage01_Setup.Step01_BuildDB
• MyApp.Test.Stage01_Setup.Step02_UpgradeDB
• MyApp.Test.Stage02_Domain.Step01_TestMyStaff
• MyApp.Test.Stage03_Integration.Step01_TestMyStaff

where MyApp.Test.Stage01_Setup is a namespace and Step01_BuildDB is a class name.