Dynamically control order of tests with pytest

Question

I would like to control the order of my tests using logic which will reorder them on the fly, while they are already running.

My use case is this: I am parallelizing my tests with xdist, and each test uses external resources from a common and limited pool. Some tests use more resources than others, so at any given time when only a fraction of the resources are available, some of the tests have the resources they need to run and others don't.

I want to optimize the usage of the resources, so I would like to dynamically choose which test will run next, based on the resources currently available. I would calculate an optimal ordering during the collection stage, but I don't know in advance how long each test will take, so I can't predict which resources will be available when.

I haven't found a way to implement this behavior with a plugin because the collection stage seems to be distinct from the running stage, and I don't know another way to modify the list of tests to run other than the collection hooks.

Would very much appreciate any suggestions, whether a way to implement this or an alternative idea which would optimize the resources utilization. My alternative is to write my own simplistic test runner, but I don't want to give up on the rest of what pytest offers.

Thanks!

Answer 1

Not really an full answer to your problem, but to your question on pytest, and few hints.

To change the order of the tests in pytest (just pytest, not pytest-xdist), you can just alter the order of the test items on the go by installing this hook wrapper:

conftest.py:

import pytest
import random

random.seed()

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_protocol(item, nextitem):
    yield

    idx = item.session.items.index(item)
    remaining = item.session.items[idx+1:]
    random.shuffle(remaining)
    item.session.items[idx+1:] = remaining

It makes no sense to change what was already executed, but only what remains — hence, [idx+1:]. In this example, I just shuffle the items, but you can do whatever you want with the list of the remaining features.

Keep in mind: This can affect how the fixtures are setup'ed & teardown'ed (those of scope above 'function'). Originally, pytest orders the tests so they can utilise the fixtures in a most efficient way. And specifically, the nextitem argument is used internally to track if the fixture should be finished. Since you efficiently change it every time, the effects can be unpredictable.

---

With pytest-xdist, it all goes totally different. You have to read on how pytest-dist distributes the tests across the slaves.

First, every slave collects all the same tests and exactly in the same order. If the order is different, pytest-xdist will fail. So you cannot reorder them on collection.

Second, the master process sends the test indexes in that collected list as the next tests to execute. So, the collection must be unchanged all the time.

Third, you can redefine the pytest_xdist_make_scheduler hook. There are few sample implementations in the pytest-xdist itself. You can define your own logic of scheduling the tests in schedule() method, using the nodes added/removed and tests collected via the corresponding methods.

Fourth, that would be too easy. The .schedule() method is called only in slave_collectionfinish event sent from the slave. I'm sad to say this, but you will have to kill and restart the slave processes all the time to trigger this event, and to re-schedule the remaining tests.

As you can see, the pytest-xdist's implementation will be huge & complex. But I hope this will give you some hints where to look at.