Proposal: control test distribution in pytest-xdist

[nicoddemus]

Here's a proposal on how to control how tests are distributed among slaves in pytest-xdist in the --dist=load mode, feature that has been requested by a lot of people in pytest-dev/pytest#175.

Note: the names here are just an initial thought, suggestions are more than welcome.

Scenarios

People have requested a number of scenarios. Here I will summarize the more popular ones and show a proposed API to support each one.

Overview

pytest-xdist in --dist=load will split tests over all slaves, where each test will run in exactly one slave distributing the test load across multiple processes and/or remotes.

Serial execution of tests of the same Class

The most popular scenario by far, people would like to somehow mark a class or module so all tests from that class/module would execute in the same slave.

import pytest

@pytest.mark.xdist_same_slave
class TestFoo:

    def test_1(self): 
        pass
    def test_2(self): 
        pass

@pytest.mark.xdist_same_slave
class TestBar:

    def test_3(self): 
        pass
    def test_4(self): 
        pass

The @pytest.mark.xdist_same_slave decorator when applied to a class will instruct xdist to run all tests of that class in the same slave. Which slave will be assigned to run tests from each class will be determined at runtime in a round-robin fashion, so as test items from classes with this decorator ar scheduled for execution, they will be assigned to execute on the same node, but different classes will have their tests assigned to different nodes. In the example above with -n 2 pytest-xdist will executes test from TestFoo in slave 0 and tests from TestBar in slave 1. The actual slave numbers are not guaranteed to be the same between runs and will be determined at runtime, with xdist doing its best to distribute tests to available slaves while keeping the constraint of binding marked tests to run in the same slave.

As tests execute in the same slave, their execution will be effectively serial.

Serial execution of tests of the same Module

Similar to the above, but will serialize all tests in a module, both contained in test classes or free functions:

import pytest

pytestmark = pytest.mark.xdist_same_slave

class Test:

    def test_1(self): 
        pass
    def test_2(self): 
        pass

def test_a():
    pass

Serial execution of tests which use a fixture

import xdist, pytest

@xdist.xdist_same_slave
@pytest.fixture(scope='session')
def db_session():
    return DBSession()

@pytest.yield_fixture
def db(db_session):
    db_session.begin_transaction()
    yield 
    db_session.end_transaction()

This will make all tests which use db execute in the same slave (which one exactly to be determined at runtime) because db depends on db_session, which is bound to execute in a single slave. This will effectively setup the database in a single slave once, and all tests which use db will run in that slave.

Unfortunately we can't use pytest.mark with fixtures, so we have to rely on a normal decorator.

---

Opinions and suggestions are more than welcome!

[nchammas]

Hey @nicoddemus, thanks for taking a crack at this. I was just getting started myself on a proposal to solve pytest-dev/pytest#175.

I wonder if there is a way to cover the common use cases while avoiding adding new options and APIs. Basically, after carefully reading through the history on pytest-dev/pytest#175, my position is now the same as what @hpk42 described in this comment:

I suggest to think how we can enhance behaviour by default, in a "no new option/API" sense as this brings the benefits to everyone.

We should refactor xdist to formalize the concept of a "chunk" of tests--i.e. tests that are always sent to the same node. Initially, every test will be its own chunk, which just yields the behavior we have today. But once we have that internal concept, we can then pass around additional information about what fixtures each test use. This information can automatically be used to chunk tests appropriately, as @hpk42 described in his comment.

If this counter-proposal sounds attractive -- not necessarily as the one we want to go with, but one that should be developed further -- then I am ready to write up a more detailed proposal as well as a candidate implementation. I have the time to iterate and see through a complete solution this month.

[nchammas]

The really attractive thing about @hpk42's proposal, apart from being "zero-config", is that it reuses the concept of a fixture to naturally chunk tests.

If two tests must run on the same slave, then presumably there is a shared resource they rely on. I think it makes sense then to ask the user to express that shared resource as a fixture, if they haven't already. From there, xdist can handle everything automatically.

[nicoddemus]

Hey @nicoddemus, thanks for taking a crack at this.

No problem. Actually I had that draft sitting around for a few months and never got around to publish it/start implementing something. The recent discussions on this topic encouraged me to go ahead and just publish it to see what feedback it would bring. 😄

We should refactor xdist to formalize the concept of a "chunk" of tests--i.e. tests that are always sent to the same node.

Great idea!

If two tests must run on the same slave, then presumably there is a shared resource they rely on. I think it makes sense then to ask the user to express that shared resource as a fixture, if they haven't already. From there, xdist can handle everything automatically.

Yep, that makes sense to me as well. Fixtures are a natural way to express dependency, plus they have rules already in place regarding inheritance and overwriting. It should be easy to gather from test items at collection time which fixtures are used by each test and their scopes, and send that information back to the master along with test ids.

If this counter-proposal sounds attractive -- not necessarily as the one we want to go with, but one that should be developed further -- then I am ready to write up a more detailed proposal as well as a candidate implementation. I have the time to iterate and see through a complete solution this month.

Please do! Feel free to write the proposal first if you want to discuss some design or implementation details before diving into the implementation. I'm more than happy to help wherever I can too. 😄

[nchammas]

Great! I'll open a new issue here with a more detailed proposal and we can take it from there.

[hpk42]

for now just two meta comments from my side: it might even make sense to create a version-controled doc (in the xdist repo eg.) where we do pull requests on. It would become the documentation then. Another thing i'd like to suggest to substitute "worker" for "slave", at least in the future documentation. the hooks are not using "slave" in their naming although we have slaveinput for transfering additional information to workers. But that's easy to alias to "worker_input" or so.

[hpk42]

another meta comment: if we have a full proposal and if there is interest we could ask people for crowd-funding a part of the likely substantial number of hours implementation will take to get to a smooth new state for xdist.

[RonnyPfannschmidt]

👍

[nchammas]

@nicoddemus - I've published a counter-proposal in #18. I'd love to hear your thoughts on it.

[nicoddemus]

Let's move the discussion over #18, which is much more solid than my initial draft!