--order-group-scope for test which located in different folder

[veenone]

Hi,

I would like to know whether it's possible to make orders for the following the structure :

\src

• \test
  • \01_feature_1
    • test_feature_a.py
      • test_id_01 ---> order(1)
      • test_id_02 ---> order(2)
    • test_another_feature_a.py
      • test_id_03 ---> order(3)
      • test_id_04 ---> order(4)
  • \02_feature_2
    • test_feature_b.py
      • test_id_01 ---> order(1)
      • test_id_02 ---> order(2)
    • test_another_feature_b.py
      • test_id_03 ---> order(3)
      • test_id_04 ---> order(4)
  • \03_feature_3
    • test_feature_c.py
      • test_id_01 ---> order(1)
      • test_id_02 ---> order(2)
    • test_another_feature_c.py
      • test_id_03 ---> order(3)
      • test_id_04 ---> order(4)

I'm executing all pytest from the \test level.
how to have the execution ordered by folder, then by the ordering number from each folder?

currently I'm using --order-scope=module --order-group-scope=class, but it seems the ordering is still not following the exact order number in 1 folder

[mrbean-bremen]

This is currently not possible. Order groups only work on module and class scope, not on directory scope.

If you are ordering across modules (like in test_feature_b / test_another_feature_a) you cannot use the module order scope. If you use session scope and module order group scope, the modules will be reordered.
As a workaround, you have to adapt the order numbers inside the features to not overlap, for example by using 21, 22, 23... for feature 2 and 31, 32, 33... for feature 3.

I think that your use case is a valid option, and I will see if I can add something later. Thanks for using the plugin :)

[mrbean-bremen]

@veenone - I added a new option --order-scope-level to master that should do what you want, e.g. in your example --order-scope-level=2 should show the wanted behavior. Please check if that works for you - I will probably make a new release soon.

[veenone]

Hi @mrbean-bremen ,

Thanks for your response. I wasn't able to give you feedback on your first comment.

I'll try this out once I'm back to my work.

Does this option stackable with the other options? (order-group & order-group-scope)?

As for the new option, how the value maps to the desired structure?

[mrbean-bremen]

As the new option also defines an order scope, it cannot be used together with --order-scope, but shall work with --order-group-scope.
Please check the documentation for usage - if that is not sufficient, I'll have to adapt it...

[mrbean-bremen]

@veenone - did you find time to test this? I will wait with a new release until I get some confirmation that it works as intended.

[veenone]

@mrbean-bremen , I will try this out tomorrow and give you the feedback on the behavior

[veenone]

Hi @mrbean-bremen ,

I have tested it but it seems it still not behaving as expected.
it still doesn't take the correct order.

\src
    \test
         \01_feature_1
                test_feature_a.py
                    test_id_01 ---> order(1)
                    test_id_02 ---> order(2)
                test_another_feature_a.py
                    test_id_03 ---> order(3)
                    test_id_04 ---> order(4)
          \02_feature_2
                 test_feature_b.py
                     test_id_01 ---> order(1)
                     test_id_02 ---> order(2)
                 test_another_feature_b.py
                     test_id_03 ---> order(3)
                     test_id_04 ---> order(4)
            \03_feature_3
                  test_feature_c.py
                      test_id_01 ---> order(1)
                      test_id_02 ---> order(2)
                  test_another_feature_c.py
                       test_id_03 ---> order(3)
                       test_id_04 ---> order(4)

I execute with the following syntax :
pytest -vv --order-scope-level=2 with / wo --order-group-scope=class

the behavior is that the test doesn't start from folder 01_feature_1, but getting the other ones,

03_feature_3:test_feature_c:test_id_01
03_feature_3:test_feature_c:test_id_02
02_feature_2:test_feature_b:test_id_01
02_feature_2:test_feature_b:test_id_02

something like that

[mrbean-bremen]

Ok, thanks - that is strange. You are calling pytest from the src folder, right? I will check if I can reproduce that tonight. Can you please copy the whole output of that test here, or are the feature 1 tests not included at all?

[veenone]

Actually I tried both calling from under src, and under test, which I know from the later one is not correct.

Do you need the screenshot of the test execution?

I think I will need to create a dummy name of them as the test information in a little bit confidential (company policy) , so I can't post it outside.

[mrbean-bremen]

If you could just try that minimal example that you posted, with empty tests (just with order markers), that would help. Something like the minimal example, with the respective output.
I will check in a few hours (still on my day job).

[mrbean-bremen]

Ok, I have tested it again, exactly with your layout described above, and it works both from src with --order-scope-level=2, and from test with order-scope-level=1. So something must be different in your setup. Have you tried the same with a minimal example?

[veenone]

Ah I found the difference actually.

I have tried the minimal example and yes it works properly.
However after comparison, my test implementation, they are using unitTest format (with class).

here's the result :

and this is the test scripts for your reference
src.zip

[mrbean-bremen]

That is strange. If I take your tests (minus the conftest.py files, as they reference some of your libraries), I get:

(pyfakefs-3.5) c:\dev\so\test_order\src>pytest -v tests --order-scope-level=2
================================================= test session starts =================================================
platform win32 -- Python 3.5.4, pytest-6.1.2, py-1.9.0, pluggy-0.13.1 -- c:\dev\venv\pyfakefs-3.5\scripts\python.exe
cachedir: .pytest_cache
rootdir: c:\dev\so\test_order\src
plugins: dependency-0.5.1, forked-1.3.0, mock-3.3.1, xdist-1.27.0, order-1.0.0.dev0
collected 16 items

tests/03_feature3/test_feature3_1.py::Test_feature3_1::test_case_3_1 PASSED                                      [  6%]
tests/03_feature3/test_feature3_1.py::Test_feature3_1::test_case_3_2 PASSED                                      [ 12%]
tests/03_feature3/test_feature3_1.py::Test_feature3_1::test_case_3_3 PASSED                                      [ 18%]
tests/03_feature3/test_feature3_1.py::Test_feature3_1::test_case_3_4 PASSED                                      [ 25%]
tests/03_feature3/test_feature3_other.py::Test_feature3_other::test_case_3_other1 PASSED                         [ 31%]
tests/03_feature3/test_feature3_other.py::Test_feature3_other::test_case_3_other2 PASSED                         [ 37%]
tests/03_feature3/test_feature3_other.py::Test_feature3_other::test_case_3_other3 PASSED                         [ 43%]
tests/03_feature3/test_feature3_other.py::Test_feature3_other::test_case_3_other4 PASSED                         [ 50%]
tests/04_feature4/test_feature4_1.py::Test_feature4_1::test_case_4_1 PASSED                                      [ 56%]
tests/04_feature4/test_feature4_1.py::Test_feature4_1::test_case_4_2 PASSED                                      [ 62%]
tests/04_feature4/test_feature4_1.py::Test_feature4_1::test_case_4_3 PASSED                                      [ 68%]
tests/04_feature4/test_feature4_1.py::Test_feature4_1::test_case_4_4 PASSED                                      [ 75%]
tests/04_feature4/test_feature4_other.py::Test_feature4_other::test_case_4_other_1 PASSED                        [ 81%]
tests/04_feature4/test_feature4_other.py::Test_feature4_other::test_case_4_other_2 PASSED                        [ 87%]
tests/04_feature4/test_feature4_other.py::Test_feature4_other::test_case_4_other_3 PASSED                        [ 93%]
tests/04_feature4/test_feature4_other.py::Test_feature4_other::test_case_4_other_4 PASSED                        [100%]

================================================= 16 passed in 0.24s ==================================================

As you can see, I used an enviroment with the same Python and pytest versions as you did, but it worked for me (I omitted the order group scope, as it doesn't make a difference in this case).

The classes shouldn't make a difference, as they are only handled after the scope has been applied - so I'm still stumped...

[mrbean-bremen]

Actually if I use module scope for your example, it also works for me. In your case, pytest seems to collect the tests in another order from the beginning. Can you make one other test - remove pytest-order, and check how the tests are run in this case? Maybe I somehow rely on the usual order that tests are run (that shouldn't be the case, but I also have not tested this). I don't see any plugin you use that changes the standard order, but something has to be different...

[mrbean-bremen]

I also notice that you use a top-level pytest.ini which you did not provide - is there anything in there that could change the test order (I doubt it, but just in case...)?

[mrbean-bremen]

Ok, while checking all plugins that are installed in your test, I found that pytest-depends changes the order the tests are executed, even if there are no dependencies defined... this explains why the tests start with feature 4 instead if feature 3, though it still does not explain the behavior.
I consider this a bug in pytest-depends - actually there is already a bug filed for this. It would make sense to remove this plugin, if you don't use it, as it needlessly interfers with other plugins that change the test order. The same functionality is also available in pytest-order.
Also, it seems to behave differently in different environments - I get different sort orders in Python 3.5 and 3.7 (sidenote: Python 3.5 has reached it's end-of-life half a year ago, so you probably should upgrade your Python version).

[veenone]

hi @mrbean-bremen , thanks for your investigation.

I have removed pytest-depends on my environment but it seems the behavior is the same :

Additionally the top level of pytest.ini (at the same level with src) doesn't contain anything, it's just a template I made before..
pytest.txt

[mrbean-bremen]

Ok, thank you - can you also make the test without pytest-order, please, to check how the sorting would be without it?

[veenone]

few seconds update!
I removed this unused pytest.ini from the folder and it changes the behavior!

now it behaves as expected!

just what made these behavior actually?

I guess your implementation is correct :) 👍

a very important note I think. not to have pytest.ini in the root folder

[veenone]

this is what I got without pytest-order installed (with pytest.ini exists at the same level as src):

and this is without pytest-order installed (without pytest.ini at the same level as src) :

[mrbean-bremen]

Thanks - I will have a look tonight!

[mrbean-bremen]

Hm - as far as I can see, you don't need to order the tests at all, they are already ordered as needed by default. I can see that your pytest.ini suppresses any warnings but does nothing else, so this shouldn't be a problem at all. I could not reproduce the behavior you had no matter what I do - installing pytest-depends or pytest-randomly merely changes the order of the features, but keeps the features together, as it should, so I still have no idea... At least it seems to work for you now, but it obviously can be broken some way.
Please keep me informed in case the problem shows up again - I would really like to understand that!

[veenone]

on the last condition, I don't have pytest-depends anymore.
I will get back to you if I found the problems on different test setup later.
For now I think your implementation is fine and can be released :)

Thank you @mrbean-bremen !