Issues counting passing rates on tests which use subTest()

[dvrogozh]

As we discussed in #34723 (comment), we spotted that tests using with self.subTest along with self.skipTest() report confusing passing rates. We need to understand how we should handle this. Below is breakdown of behavior on few syntetic tests.

Note:

• Behavior depends on whether pytest-subtest is installed or not
• Accelerate brings in pytest-subtest into environment, see https://github.com/huggingface/accelerate/blob/c0552c9012a9bae7f125e1df89cf9ee0b0d250fd/setup.py#L25
• Transformers do not depend on pytest-subtest at the moment
• Transformers ci does not have pytest-subtest in the environment at the moment (according to below breakdown this means that if first subtest is skipped other subtests won't be executed)

Synthetic tests:

• ex0.py - subTest() without skipping:

$ cat ex0.py
import unittest
class T(unittest.TestCase):
    def test_foo(self):
        for i in range(7):
            with self.subTest(i=i):
                self.assertLess(i, 3)

• ex1.py - subTest() with skipping:

$ cat ex1.py
import unittest
class T(unittest.TestCase):
    def test_foo(self):
        for i in range(7):
            with self.subTest(i=i):
                if i < 3:
                    self.skipTest(f"bon {i}")
                self.assertLess(i, 3)

		Failed	Passed	Skipped
No pytest-subtests	ex0.py	1	0	0
No pytest-subtests	ex1.py	0	0	1
With pytest-subtests	ex0.py	4	1	0
With pytest-subtests	ex1.py	1	1	3

Logs:

• With pytest-subtests:

$ python -m pytest ex0.py
...
(i=3) SUBFAIL ex0.py::T::test_foo - AssertionError: 3 not less than 3
(i=4) SUBFAIL ex0.py::T::test_foo - AssertionError: 4 not less than 3
(i=5) SUBFAIL ex0.py::T::test_foo - AssertionError: 5 not less than 3
(i=6) SUBFAIL ex0.py::T::test_foo - AssertionError: 6 not less than 3
=================================== 4 failed, 1 passed in 0.11s ===================================

$ python -m pytest ex1.py
...
(i=6) SUBFAIL ex1.py::T::test_foo - AssertionError: 6 not less than 3
============================= 1 failed, 1 passed, 3 skipped in 0.11s ===

• No pytest-subtests:

$ python -m pytest ex0.py
FAILED ex0.py::T::test_foo - AssertionError: 3 not less than 3
======================================== 1 failed in 0.11s ========================================


$ python -m pytest ex1.py
...
======================================= 1 skipped in 0.07s ========================================

CC: @ydshieh @ArthurZucker

[ydshieh]

It turns out this is related to pytest interacting with unittest and subTest/skipTest.

If running with python3 ex1.py (or ex0.py) with those files with ending

if __name__ == "__main__":
    unittest.main()

then subTest/skipTest works well together ...

[dvrogozh]

There are couple open issues which probably worth mentioning:

• Support for Python 3.4 unittest subtests pytest-dev/pytest#1367 - it's opened, long thread, last update in 2021
• self.skipTest does not produce the same results as unittest does pytest-dev/pytest-subtests#5 - it's also opened, it partially describes concerns we have, last commented 2023, no replies from maintainers

[ydshieh]

Opening an pull request in

pytest-dev/pytest-subtests#169

Might take some time to be merged (if it happens)

[ydshieh]

Closed as pytest-dev/pytest-subtests#169 is merged