Reorder test execution based on dependencies?

[jwineinger]

Is there any intent to reorder test execution according to dependencies? The current behavior I observe is that if I have two tests, where the first depends on the second, the first is just skipped. It would be preferable to reorder them so that they can both execute, rather than relying on source code order.

import pytest


@pytest.mark.dependency(depends=["test_parent"])
def test_child():
    pass


@pytest.mark.dependency
def test_parent():
    pass

$ pytest
==== test session starts ====
platform linux -- Python 3.6.4, pytest-3.5.0, py-1.5.3, pluggy-0.6.0 -- /usr/local/bin/python
cachedir: .pytest_cache
rootdir: /tests, inifile:
plugins: dependency-0.3.2
collected 2 items

test_file.py::test_child SKIPPED
test_file.py::test_parent PASSED

==== 2 passed, 1 skipped in 0.01 seconds ====

NOTE: https://github.com/ftobia/pytest-ordering does some of this, but requires explicit ordering. I'd rather just define the dependencies and let the system figure it out.

[jwineinger]

I took a stab at this problem separately from this package. https://gist.github.com/jwineinger/e41473cd5500554a189d8c97ac28f883 does work to reorder dependencies and does so across files and classes as well. I couldn't get it to play nice with the result tracking/skipping in this package but I'm intending to run pytest with -x anyway, so I don't care so much about that part.

[Formartha]

@jwineinger - seems like what I'm looking for. were you able to link tests together?

[jwineinger]

@Formartha if by link you mean specify dependencies and reorder so that dependencies are run first, then yes. To be clear though, my linked gist is separate from pytest-dependency, so you don't get any of the features it provides. Just drop the conftest.py into your top-level test directory, install deps, and decorate your tests.

[TheErk]

I have a similar need, make test run in dependency order.

There is another module trying this as well: https://github.com/ftobia/pytest-ordering
He can do ordering but not "relative dependency" as pytest dependency do.

[exhuma]

I just ran into the same issue. I have a bunch of unit tests and integration tests.
Obviously, if a unit-test fails, the corresponding integration test will fail as well but I am only directly interested in the unit-test as it will usually have a simpler more precise setup. This makes it easier to identify the problem.

The integration tests have a much more complex setup and usually have much more output as well in the failure reports. So I would love to be able to skip them when the linked unit-test fails.

But as the tests are run in a different order (I'm not going to rename my tests to control ordering) this will currently always skip the integration test. Even if the unit-test passes.

[ftesser]

Hi all, I think reordering of tests with dependencies is very useful. I used @jwineinger gist code to achieve this, I think it should be very nice to embed that idea in pytest-dependency.

[DevilXD]

I've put together a conftest.py hook that should reorder the dependent tests based on their dependencies. It even leaves you a RuntimeWarning in case you'd mistype a dependency name (I know that using node IDs can be tricky sometimes).

https://gist.github.com/DevilXD/afc7f923049a5abedea55ba186e7219c

Note that there's no cyclic dependencies validation here, and using this with those will most likely cause pytest to hang at the collecting phase. I'm leaving this here in case anyone would like to use it, and push this issue forward, at least a tiny bit.

EDIT: Updated to ignore parametrization, and properly detect dependency cycles.

[axel-h]

Will there be a new release of pytest-dependency after 0.5.1 that fixed this? Seems this issue affects many people, see for example https://stackoverflow.com/questions/17571438/test-case-execution-order-in-pytest#comment116177289_46393615

[mrbean-bremen]

FWIW: I added an option to pytest-order (a fork of pytest-ordering) to also order dependencies if needed (use --order-dependencies if the plugin is installed). For the time being, this can be used to work around this issue.

[mrbean-bremen]

Actually I think it is not a bad thing that pytest-dependency does not order the tests. I've meanwhile read a couple of comments by @RKrahl that state that he would prefer for the plugin to stay small and focused on one task, and I agree with that philosophy. Ordering tests is another task which can be accomplished using a separate plugin. I adapted pytest-order specifically to play nice with pytest-dependency, so that the combination should work ok (provided there are no bugs in pytest-order, of course).

[DevilXD]

This makes pytest-order superior to this library then, pretty sure - unless someone would really like to skip tests when others fail, I guess. Given that it's maintainer has neglected at the very least responding with their thoughts, to this issue and to the ordering PR I did...

I'm considering switching to pytest-order at this point, because @mrbean-bremen clearly appears to care more about functionality like this. I shall give it a shot.

[mrbean-bremen]

@DevilXD - as I wrote, I think the plugins complement each other. If you only need ordering, you are probably better of with pytest-order or similar, as pytest-dependency is not about ordering. That doesn't mean that pytest-order is superior in any way - it just has different goals. Also note that pytest-order is pretty new (at least the part it hasn't inherited from pytest-ordering, which is the relevent part here) and likely still has bugs...

[ftesser]

Hi all, I suggest to all the people interesting in this issue, to have a look to pytest-depends (https://pypi.org/project/pytest-depends/; https://gitlab.com/MaienM/pytest-depends).

[lukjak]

Pytest-dependecy is grossly incomplete without test ordering. It actually introduces to pytest a new logical execution order, but without any engine to enforce it. "Just" skipping is implemented.
Adding another plugin to the mix would add another level of ordering (pytest -> pytest-dependency->pytest-ordering) without any guarantee, that pytest-dependency order would not conflict with pytest-ordering one.

[RKrahl]

Pytest-dependecy is grossly incomplete without test ordering. It actually introduces to pytest a new logical execution order, but without any engine to enforce it. "Just" skipping is implemented. Adding another plugin to the mix would add another level of ordering (pytest -> pytest-dependency->pytest-ordering) without any guarantee, that pytest-dependency order would not conflict with pytest-ordering one.

pytest-dependency was never meant to be about ordering. It's all about (conditionally) skipping of tests. This may have been considered a deficiency while pytest-order was not available. But now that we have pytest-order, this issue is solved. If you believe that this solution is not appropriate and want to advance the discussion, you should come up with a concrete example where the solution does not work properly.

[lukjak]

Using both plugins forces to define order twice (implicitly with pytest-dependency and explicitly with pytest-order), while pytest-dependency already determines the execution order. Plus, it's easy to create conflicting orders:

@pytest.mark.depency(name="test1")
@pytest.mark.order(2)
def test_test1():
   pass

@pytest.mark.depency(depends="test1")
@pytest.mark.order(1)
def test_test2():
   pass

Pytest-dependency already defines execution order, but does not enforce it in any way. It should be essential part of it.

Looking at a car, all a driver does is turning a steering wheel and pushing pedals in order to travel. But there is a lot more required in addition to supporting the steering wheel and the pedals in order to make the driver travel.

[Jerther]

Pytest runs unit tests in an arbitray order. It's usually alphebetical but this cannot be relied on.

How can this plugin alone even work? I'm with @lukjak and @DevilXD on this.

[LiorPelet]

Is there any way to run @mark.depends without changing the tests order?

I want that my test will still run in the order they appear in the class but that some of the tests will be dependent on others,
For example I have this tests:

1. test_login_with_wrong_user_name
2. test_correct_login

@pytest.mark.depends(on=["test_correct_login"]
3. test_verify_user_name_after_login

In this example I want the tests will run in this order but if test_correct_login fails I don't want that test no' 3 will run.
What actually happens, when I'm running tests from some classes he mixes the run order between all the classes and making test to fail because now they running in some random order

[DevilXD]

@LiorPelet pytest-dependency currently does no test reordering, and this issue is all about adding said reordering to the plugin. If your tests are getting reordered, then it's not because of this plugin. Even with the reordering implemented, dependent tests would always run after the tests they depend on, so you wouldn't get this issue in either case.

I'd recommend taking a look at your pytest env to see if there isn't any other plugin that'd reorder your tests instead.

[mrbean-bremen]

@LiorPelet - you seem to use pytest-depends, not pytest-dependency. pytest-depends always reorders tests in an unpredictable manner, even without any dependencies defined. There is already a bug for this filed, but it looks like it will not be fixed.
That being said, pytest-depends does reorder the tests in a manner that the dependencies are executed in the correct order, it just does so in an unpredictable manner.
If you want your tests not been reordered (e.g. they are already in the correct order), you can use pytest-dependency instead. Note that you have to uninstall pytest-depends in this case, as that always reorders the tests, regardless of the markers.

[red8888]

FWIW: I added an option to pytest-order (a fork of pytest-ordering) to also order dependencies if needed (use --order-dependencies if the plugin is installed). For the time being, this can be used to work around this issue.

@mrbean-bremen I'm confused about how --order-dependencies work. Does this mean I just have to install both plugins, specify my dependencies and set --order-dependencies and it will work? Or will I have to annotate all the tests with both @pytest.mark.order and @pytest.mark.depency ? Also is --order-dependencies configured in project.toml or setup.cfg?

[mrbean-bremen]

Does this mean I just have to install both plugins, specify my dependencies and set --order-dependencies and it will work?

Yes, exactly.

You can configure it in pytest.ini.