Controlling execution order of unit tests in Visual Studio

Question

Okay, I\'m done searching for good information on this. I have a series of Unit Tests that call a static class which, once initialized, sets properties that cannot (or I don

Answer 1

As you should know by now, purists say it's forbiden to run ordered tests. That might be true for unit tests. MSTest and other Unit Test frameworks are used to run pure unit test but also UI tests, full integration tests, you name it. Maybe we shouldn't call them Unit Test frameworks, or maybe we should and use them according to our needs. That's what most people do anyway.

I'm running VS2015 and I MUST run tests in a given order because I'm running UI tests (Selenium).

Priority - Doesn't do anything at all This attribute is not used by the test system. It is provided to the user for custom purposes.

orderedtest - it works but I don't recommend it because:

1. An orderedtest a text file that lists your tests in the order they should be executed. If you change a method name, you must fix the file.
2. The test execution order is respected inside a class. You can't order which class executes its tests first.
3. An orderedtest file is bound to a configuration, either Debug or Release
4. You can have several orderedtest files but a given method can not be repeated in different orderedtest files. So you can't have one orderedtest file for Debug and another for Release.

Other suggestions in this thread are interesting but you loose the ability to follow the test progress on Test Explorer.

You are left with the solution that purist will advise against, but in fact is the solution that works: sort by declaration order.

The MSTest executor uses an interop that manages to get the declaration order and this trick will work until Microsoft changes the test executor code.

This means the test method that is declared in the first place executes before the one that is declared in second place, etc.

To make your life easier, the declaration order should match the alphabetical order that is is shown in the Test Explorer.

• A010_FirstTest
• A020_SecondTest
• etc
• A100_TenthTest

I strongly suggest some old and tested rules:

• use a step of 10 because you will need to insert a test method later on
• avoid the need to renumber your tests by using a generous step between test numbers
• use 3 digits to number your tests if you are running more than 10 tests
• use 4 digits to number your tests if you are running more than 100 tests

VERY IMPORTANT

In order to execute the tests by the declaration order, you must use Run All in the Test Explorer.

Say you have 3 test classes (in my case tests for Chrome, Firefox and Edge). If you select a given class and right click Run Selected Tests it usually starts by executing the method declared in the last place.

Again, as I said before, declared order and listed order should match or else you'll in big trouble in no time.

Answer 2

Since you've already mentioned the Ordered Test functionality that the Visual Studio testing framework supplies, I'll ignore that. You also seem to be aware that what you're trying to accomplish in order to test this Static Class is a "bad idea", so I'll ignore that to.

Instead, lets focus on how you might actually be able to guarantee that your tests are executed in the order you want. One option (as supplied by @gaog) is "one test method, many test functions", calling your test functions in the order that you want from within a single function marked with the TestMethod attribute. This is the simplest way, and the only disadvantage is that the first test function to fail will prevent any of the remaining test functions from executing.

With your description of the situation, this is the solution I would suggest you use.

If the bolded part is a problem for you, you can accomplish an ordered execution of isolated tests by leveraging the in built data driven test functionality. Its more complicated and feels a bit dirty, but it gets the job done.

In short, you define a data source (like a CSV file, or a database table) that controls the order in which you need to run your tests, and names of the functions that actually contain the test functionality. You then hook that data source into a data driven test, use the sequential read option, and execute your functions, in the order you want, as individual tests.

[TestClass]
public class OrderedTests
{
    public TestContext TestContext { get; set; }

    private const string _OrderedTestFilename = "TestList.csv";

    [TestMethod]
    [DeploymentItem(_OrderedTestFilename)]
    [DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV", _OrderedTestFilename, _OrderedTestFilename, DataAccessMethod.Sequential)]
    public void OrderedTests()
    {
        var methodName = (string)TestContext.DataRow[0];
        var method = GetType().GetMethod(methodName);
        method.Invoke(this, new object[] { });
    }

    public void Method_01()
    {
        Assert.IsTrue(true);
    }

    public void Method_02()
    {
        Assert.IsTrue(false);
    }

    public void Method_03()
    {
        Assert.IsTrue(true);
    }
}

In my example, I have a supporting file called TestList.csv, which gets copied to output. It looks like this:

TestName
Method_01
Method_02
Method_03

Your tests will be executed in the order that you specified, and in normal test isolation (i.e. if one fails, the rest still get executed, but sharing static classes).

The above is really only the basic idea, if I were to use it in production I would generate the test function names and their order dynamically before the test is run. Perhaps by leveraging PriorityAttribute you found and some simple reflection code to extract the test methods in the class and order them appropriately, then write that order to the data source.

Answer 3

You can Use Playlist

Right click on the test method -> Add to playlist -> New playlist

the execution order will be as you add them to the playlist but if you want to change it you have the file