Execution order of python unitests by their declaration

Question

I\'m using python unittests and selenium and in my code I have one test class with many testcases:

class BasicRegression(unittest.TestCase):

    @classmetho         


        

Answer 1

To start with unit tests are supposed to be independent. So must be python-unittest. Tests executed through python-unittest should be designed in such a way that they should be able to be run independently. Pure unit tests offer a benefit that when they fail, they often depicts what exactly went wrong. Still we tend to write functional tests, integration tests, and system tests with the unittest framework and these tests won't be feasible to run without ordering them since Selenium automates the Browsing Context. To achieve the ordering, you at-least need to use a better naming convention for the testnames, as an example: test_1, test_2, test_3, etc and this works because the tests are sorted respect to the built-in ordering for strings.

However, as per your observation the problem appears with test_10 and so on where sorting order seems to break. As an example, among 3 tests with name as test_1, test_2 and test_10, it seems unittest executes test_10 before test_2:

• Code:

  import unittest
  
  class Test(unittest.TestCase):
  
      @classmethod
      def setUp(self):
          print("I'm in setUp")
  
      def test_1(self):
          print("I'm in test 1")
  
      def test_2(self):
          print("I'm in test 2")
  
      def test_10(self):
          print("I'm in test 10")
  
      @classmethod
      def tearDown(self):
          print("I'm in tearDown")
  
  if __name__ == "__main__":
      unittest.main()
  

• Console Output:

  Finding files... done.
  Importing test modules ... done.
  
  I'm in setUp
  I'm in test 1
  I'm in tearDown
  I'm in setUp
  I'm in test 10
  I'm in tearDown
  I'm in setUp
  I'm in test 2
  I'm in tearDown
  ----------------------------------------------------------------------
  Ran 3 tests in 0.001s
  
  OK
  

---

Solution

Different solutions were offered in different discussions and some of them are as follows:

• @max in the discussion Unittest tests order suggested to set the sortTestMethodsUsing to None as follows:

  import unittest
  unittest.TestLoader.sortTestMethodsUsing = None
  

• @atomocopter in the discussion changing order of unit tests in Python suggested to set the sortTestMethodsUsing to some value as follows:

  import unittest
  unittest.TestLoader.sortTestMethodsUsing = lambda _, x, y: cmp(y, x)
  

• @ElmarZander in the discussion Unittest tests order suggested to use nose and write your testcases as functions (and not as methods of some TestCase derived class) nose doesn't fiddle with the order, but uses the order of the functions as defined in the file.

• @Keiji in the discussion Controlling the order of unittest.TestCases mentions:

sortTestMethodsUsing expects a function like Python 2's cmp, which has no equivalent in Python 3 (I went to check if Python 3 had a <=> spaceship operator yet, but apparently not; they expect you to rely on separate comparisons for < and ==, which seems much a backwards step...). The function takes two arguments to compare, and must return a negative number if the first is smaller. Notably in this particular case, the function may assume that the arguments are never equal, as unittest will not put duplicates in its list of test names.

With this in mind, here's the simplest way I found to do it, assuming you only use one TestCase class:

def make_orderer():
    order = {}

    def ordered(f):
        order[f.__name__] = len(order)
        return f

    def compare(a, b):
        return [1, -1][order[a] < order[b]]

    return ordered, compare

ordered, compare = make_orderer()
unittest.defaultTestLoader.sortTestMethodsUsing = compare

Then, annotate each test method with @ordered:

class TestMyClass(unittest.TestCase):
    @ordered
    def test_run_me_first(self):
        pass

    @ordered
    def test_do_this_second(self):
        pass

    @ordered
    def test_the_final_bits(self):
        pass

if __name__ == '__main__':
    unittest.main()

This relies on Python calling annotations in the order the annotated functions appear in the file. As far as I know, this is intended, and I'd be surprised if it changed, but I don't actually know if it's guaranteed behavior. I think this solution will even work in Python 2 as well, for those who are unfortunately stuck with it, though I haven't had a chance to test this.

If you have multiple TestCase classes, you'll need to run ordered, compare = make_orderer() once per class before the class definition, though how this can be used with sortTestMethodsUsing will be more tricky and I haven't yet been able to test this either.

For the record, the code I am testing does not rely on the test order being fixed - and I fully understand that you shouldn't rely on test order, and this is the reason people use to avoid answering this question. The order of my tests could be randomised and it'd work just as well. However, there is one very good reason I'd like the order to be fixed to the order they're defined in the file: it makes it so much easier to see at a glance which tests failed.