Optimize conftest.py code

[potiuk]

Some of our conftest.py code run during test collection (particularly selecting tests to run based on pytest marker) might be slowing down test collection a lot. Likely we can achieve a lot of collection speed-up if we optimize it.

[ashb]

To whoever picks this up: Some of the code is now in test_common/pytest_plugin.py -- (runs as same time, just not in conftest.py directly anymore)

[ashb]

https://github.com/zupo/awesome-pytest-speedup has some useful tips

[ashb]

Yup, conftest def makes a huge difference:

root@a5bfa17bbe52:/opt/airflow# time pytest --collect-only providers/tests/ >/dev/null

real    4m15.573s
user    1m42.059s
sys     0m3.675s
root@a5bfa17bbe52:/opt/airflow# time pytest --collect-only --noconftest providers/ >/dev/null

real    1m15.592s
user    1m2.304s
sys     0m2.346s

[potiuk]

We tried it with @enisnazif at Man's Hackathon and we really could not reproduce the "--noconftest is faster" .. If anything, we noticed that it is very unreliable, and sometimes takes longer and sometimes shorter - and it's very much independent if we use or don't use conftest. And we look at the 3s / 1000 tests as a benchmark (from pytest documentation ) and it does not seem too much off the charts.

I guess we should park that one until we separate providers.

root@4aca4313d70f:/opt/airflow# time pytest --collect-only providers/tests/ >/dev/null

real	2m8.699s
user	1m48.176s
sys	0m2.447s
root@4aca4313d70f:/opt/airflow# time pytest --collect-only providers/tests/ >/dev/null

real	2m3.896s
user	1m48.006s
sys	0m1.969s
root@4aca4313d70f:/opt/airflow# time pytest --collect-only --noconftest  providers/tests/ >/dev/null

real	2m49.071s
user	2m22.759s
sys	0m2.963s
root@4aca4313d70f:/opt/airflow# time pytest --collect-only --noconftest  providers/tests/ >/dev/null

real	3m26.905s
user	2m23.507s
sys	0m2.877s

[potiuk]

Most of the slowness came from the slow running of rglob fixed in: #51223 on Mac OS in Breeze. Other than that - there does not seem to be much more speedups. Run it with py-spy quite a few times in different cases and could not see anything "worth" optimisation.