Unittest tests order

Question

How do I be sure of the unittest methods order? Is the alphabetical or numeric prefixes the proper way?

class TestFoo(TestCase):
    def test_1(self):
               


        

Answer 1

There are a number of reasons for prioritizing tests, not the least of which is productivity, which is what JUnit Max is geared for. It's sometimes helpful to keep very slow tests in their own module so that you can get quick feedback from the those tests that that don't suffer from the same heavy dependencies. Ordering is also helpful in tracking down failures from tests that are not completely self-contained.

Answer 2

http://docs.python.org/library/unittest.html

Note that the order in which the various test cases will be run is determined by sorting the test function names with respect to the built-in ordering for strings.

If you need set order explicitly, use a monolithic test.

class Monolithic(TestCase):
  def step1(self):
      ...

  def step2(self):
      ...

  def steps(self):
    for name in sorted(dir(self)):
      if name.startswith("step"):
        yield name, getattr(self, name) 

  def test_steps(self):
    for name, step in self.steps():
      try:
        step()
      except Exception as e:
        self.fail("{} failed ({}: {})".format(step, type(e), e)

Check out post for details.

Answer 3

You can disable it by setting sortTestMethodsUsing to None: http://docs.python.org/2/library/unittest.html#unittest.TestLoader.sortTestMethodsUsing

import unittest
unittest.TestLoader.sortTestMethodsUsing = None

For pure unittests, you folks are right; but for component tests and integration tests... I do not agree that you shall assume nothing about the state. What if you are testing the state. For example, your test validates that a service is auto-started upon installation. If in your setup, you start the service, then do the assertion, then you are no longer testing the state but you are testing the "service start" functionality.

Another example is when your setup takes a long time or requires a lot of space and it just becomes impractical to run the setup frequently.

Many developers tend to use "unittest" frameworks for component testing...so stop and ask yourself, am I doing unittesting or component testing.

Answer 4

I have implemented a plugin nosedep for Nose which adds support for test dependencies and test prioritization.

As mentioned in the other answers/comments this is often a bad idea, however there can be exceptions where you would want to do this (in my case it was performance for integration tests - with a huge overhead for getting into a testable state, minutes vs hours).

A minimal example is:

def test_a:
  pass

@depends(before=test_a)
def test_b:
  pass

To ensure that test_b is always run before test_a.

Answer 5

There is no reason given that you can't build on what was done in a previous test or should rebuild it all from scratch for the next test. At least no reason is usually offered but instead people just confidently say "you shouldn't". That isn't helpful.

In general I am tired of reading too many answers here that say basically "you shouldn't do that" instead of giving any information on how to best do it if in the questioners judgment there is good reason to do so. If I wanted someone's opinion on whether I should do something then I would have asked for opinions on whether doing it is a good idea.

That out of the way, if you read say loadTestsFromTestCase and what it calls it ultimately scans for methods with some name pattern in whatever order they are encountered in the classes method dictionary, so basically in key order. It take this information and makes a testsuite of mapping it to the TestCase class. Giving it instead a list ordered as you would like is one way to do this. I am not so sure of the most efficient/cleanest way to do it but this does work.

Answer 6

It seems they are executed in alphabetical order by test name (using the comparison function between strings)

Since tests in a module are also only executed if they begin with "test", what I do is put a number to order the tests:

class LoginTest(unittest.TestCase):
    def setUp(self):
        driver.get("http://localhost:2200")

    def tearDown(self):
        # self.driver.close()
        pass
    def test1_check_at_right_page(self):
        ...
        assert "Valor" in driver.page_source

    def test2_login_a_manager(self):
        ...
        submit_button.click()
        assert "Home" in driver.title

    def test3_is_manager(self):
        ...

Note that numbers are not necessarily alphabetical - "9" > "10" in the python shell is True for instance. Consider using decimal strings with fixed 0 padding (this will avoid the aforementioned problem) such as "000", "001", ... "010"... "099", "100", ... "999".