CI Failures Associated with Pytest v8.0.0 Release

[andersy005]

What is your issue?

A recent release of pytest (v8.0.0) appears to have broken our CI.

pytest                    8.0.0              pyhd8ed1ab_0    conda-forge
pytest-cov                4.1.0              pyhd8ed1ab_0    conda-forge
pytest-env                1.1.3              pyhd8ed1ab_0    conda-forge
pytest-github-actions-annotate-failures 0.2.0                    pypi_0    pypi
pytest-timeout            2.2.0              pyhd8ed1ab_0    conda-forge
pytest-xdist              3.5.0              pyhd8ed1ab_0    conda-forge

Strangely, the issue doesn't seem to occur when using previous versions (e.g. v7.4.4). our last successful CI run used pytest v7.4.4

pytest                    7.4.4              pyhd8ed1ab_0    conda-forge
pytest-cov                4.1.0              pyhd8ed1ab_0    conda-forge
pytest-env                1.1.3              pyhd8ed1ab_0    conda-forge
pytest-github-actions-annotate-failures 0.2.0                    pypi_0    pypi
pytest-timeout            2.2.0              pyhd8ed1ab_0    conda-forge
pytest-xdist              3.5.0              pyhd8ed1ab_0    conda-forge

i recreated the environment and successfully ran tests locally. the CI failures appear to be connected to the latest release of pytest. i haven't had a chance to do an in-depth exploration of the changes from pytest which could be influencing this disruption. so, i wanted to open an issue to track what is going on. in the meantime, i'm going to pin pytest to an earlier version.

any insights, especially from those familiar with changes in the pytest v8.0.0 update, are warmly welcomed.

[ksunden]

The biggest change which seems to be affecting the tests here is that now pytest.warns does not capture all warnings, where previously it did.

We had similar changes needed in mpl: matplotlib/matplotlib#27624

In particular, nesting pytest.raises and pytest.warns in pytest 7 would cause things to not fail even if one of those conditions were not met (e.g. the exception is raised before the warning would have been issued) so in pytest 8 this now (correctly I think) tells you when you thought you were testing that a warning was issued that is not, in fact, actually issued.

A quick scan of the failing tests looked like all of them had that pattern of nested raises and warns. The fix is either to ensure that expected warnings are issued before exceptions interrupt control flow or to not test for warnings that are not issued.

[mgorny]

I'm going to try making a pull request for this.