pytest-timeout - fail test instead killing whole test run

Question

I know in pytest-timeout I can specify timeout for each testcase but single failure terminates whole test run instead failing the slacking off testcase.

Am I forced to make my own solution of this or there are ready-to-use tools which provide that?

Answer 1

I looked into this issue a long time ago and also came to the conclusion that a self-made solution would be better.

My plugin was killing the whole pytest process, but it can be adjusted to fail only a single (current) test easily. Here is the adjusted draft:

import pytest
import signal


class Termination(SystemExit):
    pass


class TimeoutExit(BaseException):
    pass


def _terminate(signum, frame):
    raise Termination("Runner is terminated from outside.")


def _timeout(signum, frame):
    raise TimeoutExit("Runner timeout is reached, runner is terminating.")


@pytest.hookimpl
def pytest_addoption(parser):
    parser.addoption(
        '--timeout', action='store', dest='timeout', type=int, default=None,
        help="number of seconds before each test failure")


@pytest.hookimpl
def pytest_configure(config):
    # Install the signal handlers that we want to process.
    signal.signal(signal.SIGTERM, _terminate)
    signal.signal(signal.SIGALRM, _timeout)


@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_protocol(item, nextitem):

    # Set the per-test timeout (an alarm signal).
    if item.config.option.timeout is not None:
        signal.alarm(item.config.option.timeout)

    try:
        # Run the setup, test body, and teardown stages.
        yield
    finally:
        # Disable the alarm when the test passes or fails.
        # I.e. when we get into the framework's body.
        signal.alarm(0)

---

When you do kill -ALRM $pid, or when each test times out individually due to the preset alarm, only the current test will fail, but the other tests will continue.

And this TimeoutExit will not be suppressed by the libraries which do except Exception: pass because it inherits from BaseException.

So, it is alike SystemExit in this aspect. However, unlike SystemExit or KeyboardInterruption, pytest will not catch it, and will not exit on such an exception.

The exception will be injected into wherever the test does at the moment of the alarm, even if it does time.sleep(...) (so as for any signals).

Remember, that you can only have one single alarm set for the process (OS limitation). Which also makes it incompatible with pytest-timeout, because it also uses the ALRM signal for the same purpose.

If you want to have the global & per-test timeouts, you have to implement you smart alarm manager, which will keep track of few alarms, set the OS alarm to the earliest one, and decide which handler to call when the alarm signal is received.

---

In case, when you do kill -TERM $pid or just kill $pid (graceful termination), it will be terminated immediately — because it inherits from SystemExit, which is the BaseException and is usually not caught by the code or by pytest.

The latter case mostly demonstrates how you can set different reactions to different signals. You can do the similar things with USR1 & USR2 and other catchable signals.

---

For a quick-test, put the plugin code above to the conftest.py file (a pseudo-plugin).

Consider this test file:

import time

def test_this():
    try:
        time.sleep(10)
    except Exception:
        pass

def test_that():
    pass

Running pytest without a timeout does nothing, and both tests pass:

$ pytest -s -v
.........
collected 2 items                                                                                                                                                                 

test_me.py::test_this PASSED
test_me.py::test_that PASSED

======= 2 passed in 10.02 seconds =======

Running it with the timeout fail the first test, but pass the second one:

$ pytest -s -v --timeout=5
.........
collected 2 items                                                                                                                                                                 

test_me.py::test_this FAILED
test_me.py::test_that PASSED

============== FAILURES ==============
______________ test_this _____________

    def test_this():
        try:
>           time.sleep(10)

test_me.py:5: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

signum = 14, frame = <frame object at 0x106b3c428>

    def _timeout(signum, frame):
>       raise TimeoutExit("Runner timeout is reached, runner is terminating.")
E       conftest.pytest_configure.<locals>.TimeoutExit: Runner timeout is reached, runner is terminating.

conftest.py:24: TimeoutExit
======= 1 failed, 1 passed in 5.11 seconds =======

Answer 2

This has been fully supported by pytest-timeout right from the beginning, you want to use the signal method as described in the readme of pytest-timeout. Please do read the readme carefully as it comes with some caveats. And indeed it is implemented using SIGALRM as the other answer also suggests, but it already exists so no need to re-do this.