Python unittest: Retry on failure with Nose?

Question

I have a test which randomly fails and I want to let it retry a number of times before sending an error message.

I'm using python with Nose.

I wrote the following, but unfortunately, even with the try/except handling, Nose returns an error when the test fails on the first try.

def test_something(self):
    maxAttempts = 3
    func = self.run_something

    attempt = 1
    while True:
        if attempt == maxAttempts:
            yield func
            break

        else:
            try:
                yield func
                break
            except:
                attempt += 1

def run_something(self):
    #Do stuff

Thanks

Answer 1

By using a generator, you're giving nose maxAttempts tests to run. if any of them fail, the suite fails. The try/catch doesn't particularly apply to the tests your yielding, since its nose that runs them. Rewrite your test like so:

def test_something(self):
    maxAttempts = 3
    func = self.run_something

    attempt = 1
    while True:
        if attempt == maxAttempts:
            func() # <<<--------
            break

        else:
            try:
                func() # <<<--------
                break
            except:
                attempt += 1

def run_something(self):
    #Do stuff

Answer 2

You can use attributes on your functions with the flaky nose plugin that will automatically re-run tests and let you use advanced parameters (like if 2 in 3 test pass, then it's a pass)

GitHub flaky project

How to install Flaky plugin for Python:

pip install flaky

Example nose test runner configuration:

nosetests.exe your_python_tests.py --with-flaky --force-flaky --max-runs=3

Example Python code with function marked with Flaky attribute:

from flaky import flaky

@flaky
def test_something_that_usually_passes(self):
    value_to_double = 21
    result = get_result_from_flaky_doubler(value_to_double)
    self.assertEqual(result, value_to_double * 2, 'Result doubled incorrectly.')