Unittest tests order

Question

How do I be sure of the unittest methods order? Is the alphabetical or numeric prefixes the proper way?

class TestFoo(TestCase):
    def test_1(self):
               


        

Answer 1

Here is a simpler method that has the following advantages:

• No need to create a custom TestCase class.
• No need to decorate every test method.
• Use the unittest standard load test protocol. See the Python docs here.

The idea is to go through all the test cases of the test suites given to the test loader protocol and create a new suite but with the tests ordered by their line number.

Here is the code:

import unittest

def load_ordered_tests(loader, standard_tests, pattern):
    """
    Test loader that keeps the tests in the order they were declared in the class.
    """
    ordered_cases = []
    for test_suite in standard_tests:
        ordered = []
        for test_case in test_suite:
            test_case_type = type(test_case)
            method_name = test_case._testMethodName
            testMethod = getattr(test_case, method_name)
            line = testMethod.__code__.co_firstlineno
            ordered.append( (line, test_case_type, method_name) )
        ordered.sort()
        for line, case_type, name in ordered:
            ordered_cases.append(case_type(name))
    return unittest.TestSuite(ordered_cases)

You can put this in a module named order_tests and then in each unittest Python file, declare the test loader like this:

from order_tests import load_ordered_tests

# This orders the tests to be run in the order they were declared.
# It uses the unittest load_tests protocol.
load_tests = load_ordered_tests

Note: the often suggested technique of setting the test sorter to None no longer works because Python now sorts the output of dir() and unittest uses dir() to find tests. So even though you have no sorting method, they still get sorted by Python itself!

Answer 2

Contrary to what was said here: - tests have to run in isolation (order must not matters for that) AND - ordering them is important because they describe what the system do and how the developper implements it.

IOW, each test brings you informations of the system and the developper logic.

So if these informations are not ordered that can make your code difficult to understand.

Answer 3

If you use 'nose' and you write your test cases as functions (and not as methods of some TestCase derived class) 'nose' doesn't fiddle with the order, but uses the order of the functions as defined in the file. In order to have the assert_* methods handy without needing to subclass TestCase I usually use the testing module from numpy. Example:

from numpy.testing import *

def test_aaa():
    assert_equal(1, 1)

def test_zzz():
    assert_equal(1, 1)

def test_bbb():
    assert_equal(1, 1)

Running that with ''nosetest -vv'' gives:

test_it.test_aaa ... ok
test_it.test_zzz ... ok
test_it.test_bbb ... ok
----------------------------------------------------------------------
Ran 3 tests in 0.050s
OK

Note to all those who contend that unit tests shouldn't be ordered: while it is true that unit tests should be isolated and can run independently, your functions and classes are usually not independent. They rather build up on another from simpler/low-level functions to more complex/high-level functions. When you start optimising your low-level functions and mess up (for my part, I do that frequently; if you don't, you probably don't need unit test anyway;-) then it's a lot better for diagnosing the cause, when the tests for simple functions come first, and tests for functions that depend on those functions later. If the tests are sorted alphabetically the real cause usually gets drowned among one hundred failed assertions, which are not there because the function under test has a bug, but because the low-level function it relies on has.

That's why I want to have my unit tests sorted the way I specified them: not to use state that was built up in early tests in later tests, but as a very helpful tool in diagnosing problems.

Answer 4

See the example of WidgetTestCase on http://docs.python.org/library/unittest.html#organizing-test-code , it says that

Class instances will now each run one of the test_*() methods, with self.widget created and destroyed separately for each instance.

So it might be of no use to specify the order of test cases, if you do not access global variables.

Answer 5

The philosophy behind unit tests is to make them independent of each other. This means that the first step of each test will always be to try to rethink how you are testing each piece to match that philosophy. This can involve changing how you approach testing and being creative by narrowing your tests to smaller scopes.

However, if you still find that you need tests in a specific order (as that is viable), you could try checking out the answer to Python unittest.TestCase execution order .

Answer 6

Why do you need specific test order? The tests should be isolated and therefore it should be possible to run them in any order, or even in parallel.

If you need to test something like user unsubscribing, the test could create a fresh database with a test subscription and then try to unsubscribe. This scenario has its own problems, but in the end it’s better than having tests depend on each other. (Note that you can factor out common test code, so that you don’t have to repeat the DB setup code or create testing data ad nauseam.)