Unittest tests order

Question

How do I be sure of the unittest methods order? Is the alphabetical or numeric prefixes the proper way?

class TestFoo(TestCase):
    def test_1(self):
               


        

Answer 1

To randomise the order of test methods you can monkey patch the unittest.TestLoader.sortTestMethodsUsing attribute

if __name__ == '__main__':
    import random
    unittest.TestLoader.sortTestMethodsUsing = lambda self, a, b: random.choice([1, 0, -1])
    unittest.main()

The same approach can be used to enforce whatever order you need.

Answer 2

Ok, may be a bit later, but anyway...

You should try proboscis library. It will allow you to make tests order as well as set up any test dependencies. I use it and this library is awesome truly.

For example, if test case #1 from module A should depend on test case #3 from module B you CAN set this behaviour using the library.

Answer 3

I half agree with the idea that tests souldn't be ordered. In some cases it helps (it's easier damn it!) to have them in order... after all that's the reason for the 'unit' in UnitTest.

That said one alternative is to use mock objects to mockout and patch the items that should run before that specific code under test. You can also put a dummy function in there to monkey patch your code. For more info check out Mock, which is part of the standard library now. Mock

Here are some YouTube videos if you haven't used Mock before.

Video 1

Video 2

Video 3

More to the point, try using class methods to structure your code, then place all the class methods in one main test method.

import unittest
import sqlite3

class MyOrderedTest(unittest.TestCase):

    @classmethod
    def setUpClass(cls):
        cls.create_db()
        cls.setup_draft()
        cls.draft_one()
        cls.draft_two()
        cls.draft_three()

    @classmethod
    def create_db(cls):
        cls.conn = sqlite3.connect(":memory:")

    @classmethod
    def setup_draft(cls):
        cls.conn.execute("CREATE TABLE players ('draftid' INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, 'first', 'last')")

    @classmethod
    def draft_one(cls):
        player = ("Hakeem", "Olajuwon")
        cls.conn.execute("INSERT INTO players (first, last) VALUES (?, ?)", player)

    @classmethod
    def draft_two(cls):
        player = ("Sam", "Bowie")
        cls.conn.execute("INSERT INTO players (first, last) VALUES (?, ?)", player)

    @classmethod
    def draft_three(cls):
        player = ("Michael", "Jordan")
        cls.conn.execute("INSERT INTO players (first, last) VALUES (?, ?)", player)

    def test_unordered_one(self):
        cur = self.conn.execute("SELECT * from players")
        draft = [(1, u'Hakeem', u'Olajuwon'), (2, u'Sam', u'Bowie'), (3, u'Michael', u'Jordan')]
        query = cur.fetchall()
        print query
        self.assertListEqual(query, draft)

    def test_unordered_two(self):
        cur = self.conn.execute("SELECT first, last FROM players WHERE draftid=3")
        result = cur.fetchone()
        third = " ".join(result)
        print third
        self.assertEqual(third, "Michael Jordan")

Answer 4

There are scenarios where the order can be important and where setUp and Teardown come in as to limited. There's only one setUp and tearDown method, which is logical but you can only put so much information in them until it get's unclear what setUp or tearDown might actually be doing.

Take this integration test as an example:

You are writing tests to see if the registration form and the login form are working correctly. In such a case the order is important, as you can't login without an existing account. More importantly the order of your tests represents some kind of user interaction. Where each test might represent a step in the whole process or flow you're testing.

Dividing your code in those logical pieces has several advantages.

Might not be the best solution but, I often use one method that kicks off the actual tests.

def test_registration_login_flow(self):
    _test_registration_flow()
    _test_login_flow()

Answer 5

I agree with the statement that a blanket "don't do that" answer is a bad response.

I have a similar situation where I have a single data source and one test will wipe the data set causing other tests to fail.

My solution was to use the operating system environment variables in my Bamboo server...

(1) The test for the "data purge" functionality starts with a while loop that checks the state of an environment variable "BLOCK_DATA_PURGE." If the "BLOCK_DATA_PURGE" variable is greater than zero, the loop will write a log entry to the effect that it is sleeping 1 second. Once the "BLOCK_DATA_PURGE" has a zero value, execution proceeds to test the purge functionality.

(2) Any unit test which needs the data in the table simply increments "BLOCK_DATA_PURGE" at the beginning (in setup()) and decrements the same variable in teardown().

The effect of this is to allow various data consumers to block the purge functionality so long as they need without fear that the purge could execute in between tests. Effectively the purge operation is pushed to the last step...or at least the last step that requires the original data set.

Today I am going to extend this to add more functionality to allow some tests to REQUIRE_DATA_PURGE. These will effectively invert the above process to ensure that those tests only execute after the data purge to test data restoration.

Answer 6

A simple method for ordering "unittest" tests is to follow the init.d mechanism of giving them numeric names:

def test_00_createEmptyObject(self):
    obj = MyObject()
    self.assertIsEqual(obj.property1, 0)
    self.assertIsEqual(obj.dict1, {})

def test_01_createObject(self):
    obj = MyObject(property1="hello", dict1={"pizza":"pepperoni"})
    self.assertIsEqual(obj.property1, "hello")
    self.assertIsDictEqual(obj.dict1, {"pizza":"pepperoni"})

def test_10_reverseProperty(self):
    obj = MyObject(property1="world")
    obj.reverseProperty1()
    self.assertIsEqual(obj.property1, "dlrow")

However, in such cases, you might want to consider structuring your tests differently so that you can build on previous construction cases. For instance, in the above, it might make sense to have a "construct and veirfy" function that constructs the object and validates it's assignment of parameters.

def make_myobject(self, property1, dict1):  # Must be specified by caller
    obj = MyObject(property1=property1, dict1=dict1)
    if property1:
        self.assertEqual(obj.property1, property1)
    else:
        self.assertEqual(obj.property1, 0)
    if dict1:
        self.assertDictEqual(obj.dict1, dict1)
    else:
        self.assertEqual(obj.dict1, {})
    return obj

def test_00_createEmptyObject(self):
    obj = self.make_object(None, None)

def test_01_createObject(self):
    obj = self.make_object("hello", {"pizza":"pepperoni"})

def test_10_reverseProperty(self):
    obj = self.make_object("world", None)
    obj.reverseProperty()
    self.assertEqual(obj.property1, "dlrow")