pytest -v does not show the 'msg' contents in the report for subtests

[bskinn]

For the following test function, code and test details unimportant:

def test_apifail_inventory_no_object_invs(subtests):
    """Confirm no-objects inventories don't import."""
    inv = soi.Inventory()

    with subtests.test(msg="plain"):
        with pytest.raises(TypeError):
            soi.Inventory(inv.data_file())

    with subtests.test(msg="zlib"):
        with pytest.raises((TypeError, ValueError)):
            soi.Inventory(soi.compress(inv.data_file()))

    d = {"project": "test", "version": "0.0", "count": 0}
    with subtests.test(msg="json"):
        with pytest.raises(ValueError):
            soi.Inventory(d)

The following output is obtained; the contents of msg are not included in the line of the report for each subtest (marked with arrows):

>pytest -vk no_object
================================================= test session starts =================================================
platform win32 -- Python 3.6.3, pytest-4.4.0, py-1.8.0, pluggy-0.9.0 -- c:\temp\git\sphobjinv\env\scripts\python.exe
cachedir: .pytest_cache
rootdir: C:\Temp\git\sphobjinv, inifile: tox.ini
plugins: timeout-1.3.3, subtests-0.1.0, ordering-0.6, cov-2.6.1
collected 173 items / 172 deselected / 1 selected

tests/test_api_fail.py::test_apifail_inventory_no_object_invs PASSED     [100%]  <---
tests/test_api_fail.py::test_apifail_inventory_no_object_invs PASSED     [100%]  <---
tests/test_api_fail.py::test_apifail_inventory_no_object_invs PASSED     [100%]  <---
tests/test_api_fail.py::test_apifail_inventory_no_object_invs PASSED                                             [100%]

====================================== 1 passed, 172 deselected in 0.30 seconds =======================================

This is problematic when a subtest fails and something like --tb=line is used. After making a breaking edit to one of the subtests and rerunning:

>pytest -vk no_object --tb=line
================================================= test session starts =================================================
platform win32 -- Python 3.6.3, pytest-4.4.0, py-1.8.0, pluggy-0.9.0 -- c:\temp\git\sphobjinv\env\scripts\python.exe
cachedir: .pytest_cache
rootdir: C:\Temp\git\sphobjinv, inifile: tox.ini
plugins: timeout-1.3.3, subtests-0.1.0, ordering-0.6, cov-2.6.1
collected 173 items / 172 deselected / 1 selected

tests/test_api_fail.py::test_apifail_inventory_no_object_invs FAILED     [100%]    <===
tests/test_api_fail.py::test_apifail_inventory_no_object_invs PASSED     [100%]
tests/test_api_fail.py::test_apifail_inventory_no_object_invs PASSED     [100%]
tests/test_api_fail.py::test_apifail_inventory_no_object_invs PASSED                                             [100%]

====================================================== FAILURES =======================================================
c:\temp\git\sphobjinv\src\sphobjinv\inventory.py:626: TypeError: Invalid Inventory source type
================================= 1 failed, 1 passed, 172 deselected in 0.34 seconds ==================================

It would be nice to have the msg reported with each subtest, so as to avoid having to rerun the test with expanded context to identify offending case(s).

[nicoddemus]

Hi @bskinn,

Thanks for the report!

You are absolutely right. I'm afraid I will have to add some more things to pytest in order to be able to achieve this output though. 👍

[nicoddemus]

Hi @thedrow, unfortunately this has stalled a bit. Still trying to figure out the best approach, which might be to go ahead and integrate this into pytest rather than add further capabilities to it. Let's see. 👍

[mohamed--abdel-maksoud]

Just to say I'm also waiting for this feature!
Thanks for the awesome project, a life saver indeed.

[mauvilsa]

Created pull request #88 with a proposal for an interim solution to this.