Controlling execution order of unit tests in Visual Studio

Question

Okay, I\'m done searching for good information on this. I have a series of Unit Tests that call a static class which, once initialized, sets properties that cannot (or I don

Answer 1

Merge your tests into one giant test will work. To make the test method more readable, you can do something like

[TestMethod]
public void MyIntegratonTestLikeUnitTest()
{
    AssertScenarioA();

    AssertScenarioB();

    ....
}

private void AssertScenarioA()
{
     // Assert
}

private void AssertScenarioB()
{
     // Assert
}

Actually the issue you have suggests you probably should improve the testability of the implementation.

Answer 2

I see that this topic is almost 6 years old, and we now have new version of Visual studio but I will reply anyway. I had that order problem in Visual Studio 19 and I figured it out by adding capital letter (you can also add small letter) in front of your method name and in alphabetical order like this:

[TestMethod]
        public void AName1()
        {}
[TestMethod]
        public void BName2()
        {}

And so on. I know that this doesn't look appealing, but it looks like Visual is sorting your tests in test explorer in alphabetical order, doesn't matter how you write it in your code. Playlist didn't work for me in this case.

Hope that this will help.

Answer 3

Here is a class that can be used to setup and run ordered tests independent of MS Ordered Tests framework for whatever reason--like not have to adjust mstest.exe arguments on a build machine, or mixing ordered with non-ordered in a class.

The original testing framework only sees the list of ordered tests as a single test so any init/cleanup like [TestInitalize()] Init() is only called before and after the entire set.

Usage:

        [TestMethod] // place only on the list--not the individuals
        public void OrderedStepsTest()
        {
            OrderedTest.Run(TestContext, new List<OrderedTest>
            {
                new OrderedTest ( T10_Reset_Database, false ),
                new OrderedTest ( T20_LoginUser1, false ),
                new OrderedTest ( T30_DoLoginUser1Task1, true ), // continue on failure
                new OrderedTest ( T40_DoLoginUser1Task2, true ), // continue on failure
                // ...
            });                
        }

Implementation:

using Microsoft.VisualStudio.TestTools.UnitTesting;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;

namespace UnitTests.Utility
{    
    /// <summary>
    /// Define and Run a list of ordered tests. 
    /// 2016/08/25: Posted to SO by crokusek 
    /// </summary>    
    public class OrderedTest
    {
        /// <summary>Test Method to run</summary>
        public Action TestMethod { get; private set; }

        /// <summary>Flag indicating whether testing should continue with the next test if the current one fails</summary>
        public bool ContinueOnFailure { get; private set; }

        /// <summary>Any Exception thrown by the test</summary>
        public Exception ExceptionResult;

        /// <summary>
        /// Constructor
        /// </summary>
        /// <param name="testMethod"></param>
        /// <param name="continueOnFailure">True to continue with the next test if this test fails</param>
        public OrderedTest(Action testMethod, bool continueOnFailure = false)
        {
            TestMethod = testMethod;
            ContinueOnFailure = continueOnFailure;
        }

        /// <summary>
        /// Run the test saving any exception within ExceptionResult
        /// Throw to the caller only if ContinueOnFailure == false
        /// </summary>
        /// <param name="testContextOpt"></param>
        public void Run()
        {
            try
            {
                TestMethod();
            }
            catch (Exception ex)
            {
                ExceptionResult = ex;
                throw;
            }
        }

        /// <summary>
        /// Run a list of OrderedTest's
        /// </summary>
        static public void Run(TestContext testContext, List<OrderedTest> tests)
        {
            Stopwatch overallStopWatch = new Stopwatch();
            overallStopWatch.Start();

            List<Exception> exceptions = new List<Exception>();

            int testsAttempted = 0;
            for (int i = 0; i < tests.Count; i++)
            {
                OrderedTest test = tests[i];

                Stopwatch stopWatch = new Stopwatch();
                stopWatch.Start();

                testContext.WriteLine("Starting ordered test step ({0} of {1}) '{2}' at {3}...\n",
                    i + 1,
                    tests.Count,
                    test.TestMethod.Method,
                    DateTime.Now.ToString("G"));

                try
                {
                    testsAttempted++;
                    test.Run();
                }
                catch
                {
                    if (!test.ContinueOnFailure)
                        break;
                }
                finally
                {
                    Exception testEx = test.ExceptionResult;

                    if (testEx != null)  // capture any "continue on fail" exception
                        exceptions.Add(testEx);

                    testContext.WriteLine("\n{0} ordered test step {1} of {2} '{3}' in {4} at {5}{6}\n",
                        testEx != null ? "Error:  Failed" : "Successfully completed",
                        i + 1,
                        tests.Count,
                        test.TestMethod.Method,
                        stopWatch.ElapsedMilliseconds > 1000
                            ? (stopWatch.ElapsedMilliseconds * .001) + "s"
                            : stopWatch.ElapsedMilliseconds + "ms",
                        DateTime.Now.ToString("G"),
                        testEx != null
                            ? "\nException:  " + testEx.Message +
                                "\nStackTrace:  " + testEx.StackTrace +
                                "\nContinueOnFailure:  " + test.ContinueOnFailure
                            : "");
                }
            }

            testContext.WriteLine("Completed running {0} of {1} ordered tests with a total of {2} error(s) at {3} in {4}",
                testsAttempted,
                tests.Count,
                exceptions.Count,
                DateTime.Now.ToString("G"),
                overallStopWatch.ElapsedMilliseconds > 1000
                    ? (overallStopWatch.ElapsedMilliseconds * .001) + "s"
                    : overallStopWatch.ElapsedMilliseconds + "ms");

            if (exceptions.Any())
            {
                // Test Explorer prints better msgs with this hierarchy rather than using 1 AggregateException().
                throw new Exception(String.Join("; ", exceptions.Select(e => e.Message), new AggregateException(exceptions)));
            }
        }
    }
}

Answer 4

I dont see anyone mentioning the ClassInitialize attribute method. The attributes are pretty straight forward.

Create methods that are marked with either the [ClassInitialize()] or [TestInitialize()] attribute to prepare aspects of the environment in which your unit test will run. The purpose of this is to establish a known state for running your unit test. For example, you may use the [ClassInitialize()] or the [TestInitialize()] method to copy, alter, or create certain data files that your test will use.

Create methods that are marked with either the [ClassCleanup()] or [TestCleanUp{}] attribute to return the environment to a known state after a test has run. This might mean the deletion of files in folders or the return of a database to a known state. An example of this is to reset an inventory database to an initial state after testing a method that is used in an order-entry application.

• [ClassInitialize()] Use ClassInitialize to run code before you run the first test in the class.

• [ClassCleanUp()] Use ClassCleanup to run code after all tests in a class have run.

• [TestInitialize()] Use TestInitialize to run code before you run each test.

• [TestCleanUp()] Use TestCleanup to run code after each test has run.

Answer 5

I'll not address the order of tests, sorry. Others already did it. Also, if you know about "ordered tests" - well, this is MS VS's response to the problem. I know that those ordered-tests are no fun. But they thought it will be "it" and there's really nothing more in MSTest about that.

I write about one of your assumptions:

as there is no way to tear down the static class.

Unless your static class represents some process-wide external state external to your code (like ie. the state of an unmanaged native DLL library thats P/Invoked by the rest of your code), your assumption that there is no way is not true.

If your static class refers to this, then sorry, you are perfectly right, the rest of this anwer is irrelevant. Still, as you didn't say that, I assume your code is "managed".

Think and check the AppDomain thingy. Rarely it is needed, but this is exactly the case when you'd probably like to use them.

You can create a new AppDomain, and instantiate the test there, and run the test method there. Static data used by managed code will isolated there and upon completion, you will be able to unload the AppDomain and all the data, statics included, will evaporate. Then, next test would initialize another appdomain, and so on.

This will work unless you have external state that you must track. AppDomains only isolate the managed memory. Any native DLL will still be load per-process and their state will be shared by all AppDomains.

Also, creating/tearing down the appdomains will, well, slow down the tests. Also, you may have problems with assembly resolution in the child appdomain, but they are solvable with reasonable amount of reusable code.

Also, you may have small problems with passing test data to - and back from - the child AppDomain. Objects passed will either have to be serializable in some way, or be MarshalByRef or etc. Talking cross-domain is almost like IPC.

However, take care here, it will be 100% managed talking. If you take some extra care and add a little work to the AppDomain setup, you will be able to even pass delegates and run them in the target domain. Then, instead of making some hairy cross-domain setup, you can wrap your tests with to something like:

void testmethod()
{
    TestAppDomainHelper.Run( () =>
    {
        // your test code
    });
}

or even

[IsolatedAppDomain]
void testmethod()
{
    // your test code
}

if your test framework supports creating such wrappers/extensions. After some initial research and work, using them is almost trivial.

Answer 6

they just can't be run together in a random order as there is no way to tear down the static class

You can name namespaces and classes in alphabetical order. eg.:

• MyApp.Test.Stage01_Setup.Step01_BuildDB
• MyApp.Test.Stage01_Setup.Step02_UpgradeDB
• MyApp.Test.Stage02_Domain.Step01_TestMyStaff
• MyApp.Test.Stage03_Integration.Step01_TestMyStaff

where MyApp.Test.Stage01_Setup is a namespace and Step01_BuildDB is a class name.