unittest runner: avoid tearDown and cleanup to ease post mortem debugging

[mbyt]

Assuming pytest is used to run a GUI test unittest suite. Typically GUI tests close the GUI in the unittest tearDowns or cleanup methods.

These tests cannot be easily post mortem debugged (pytest --pdb), as unittests tearDown and cleanup (which close the GUI) are called on an exception before the post mortem debugger jumps into place. Therefore, the GUI is closed and no direct interaction is possible any more.

A work around when running the unittest via the unittest runner is to add the following line to ~/.pdbrc: !import unittest; unittest.TestCase.run = lambda self,*args,**kw: unittest.TestCase.debug(self), start the tests with python -m pdb test.py and hit continue.

This pull request accomplishes the same within the pytest unittest compatibility layer. It is solely activated if pytest ist called with --pdb.

An example program which shows the problem would be the following (run with and without this change):

from __future__ import print_function
import unittest

class Blub(unittest.TestCase):
    def setUp(self):
        self.addCleanup(lambda: print("2 Cleanup"))

    def tearDown(self):
        print("1 tearDown")

    @classmethod
    def tearDownClass(self):
        print("3 tearDownClass")

    def test_false(self):
        self.assertTrue(False)

def suite():
    return unittest.makeSuite(Blub, 'test_')

if __name__ == '__main__':
    unittest.main(defaultTest='suite')

[mbyt]

Please let me know what you think about the problem. Is there another / more simple solution to this problem?

In case the proposed solution is sensible, I'll add a corresponding test.

[coveralls]

Coverage decreased (-0.01%) to 93.017% when pulling 4eeb475 on mbyt:disable_tearDown_and_cleanups_for_post_mortem_debugging into 82218e4 on pytest-dev:master.

[nicoddemus]

Hey @mbyt, thanks for the PR.

TBH I'm reluctant of adding such an esoteric use-case to pytest core, specially on the unittest layer. It seems problematic to simply disable teardowns completely when pytest is invoked with --pdb. On the other hand, one usually doesn't use the --pdb option unless it is running a specific test which is failing and he/she wants to debug.

You can accomplish something similar by adding this to your conftest.py:

def pytest_configure(config):
    if config.getvalue("usepdb"):
         import _pytest.unittest
         _pytest.unittest.TestCaseFunction.runtest = lambda self,*args,**kw: self._testcase.debug()

[nicoddemus]

Hmm now that I thought a little more and played around a bit, I think actually we somehow should make it possible to use --pdb with unittest subclasses.

Consider:

import unittest

class Test(unittest.TestCase):

    def tearDown(self):
        self.filename = None

    def test_1(self):
        assert 1

    def test_2(self):
        self.filename = 'debug me'
        assert 0

If --pdb is used, without @mbyt's PR there's no way to evaluate in the debugger what self.filename was on the failing test_2, so it is very limited at that point for actually debugging.

One might even argue that --pdb is currently broken when used in unittest sub-classes.

What do others think?

[RonnyPfannschmidt]

i believe this is something we want to add, but it should be accompanied with a test so we dont break it

[coveralls]

Coverage decreased (-0.01%) to 93.078% when pulling be08223 on mbyt:disable_tearDown_and_cleanups_for_post_mortem_debugging into 67ba8aa on pytest-dev:master.

[mbyt]

Thank you very much for your insights and your fast reply.

I added a corresponding test.

Another possible solution would be to read the .pdbrc during the initialization, which would allow to enable monkey patches as the one described.

[nicoddemus]

Don't you want to use your real name here?

[mbyt]

Thanks for letting me know. If you do not mind, let's leave it like this.

[nicoddemus]

Thanks! 😁

Other than my comments above, would you mind adding a small blurb to the docs mentioning this change? I think adding a .. note:: to unittest.rst right before the Mixing pytest fixtures into unittest.TestCase style tests section would be great, because the previous paragraph mentions using --pdb for debugging.

[mbyt]

@nicoddemus thanks again for looking at the changesets and your constructive comments. I modified the code accordingly.

[coveralls]

Coverage decreased (-0.01%) to 93.078% when pulling 696a911 on mbyt:disable_tearDown_and_cleanups_for_post_mortem_debugging into 67ba8aa on pytest-dev:master.

[coveralls]

Coverage decreased (-0.01%) to 93.078% when pulling e43d117 on mbyt:disable_tearDown_and_cleanups_for_post_mortem_debugging into 67ba8aa on pytest-dev:master.

[nicoddemus]

Thanks again!

[blueyed]

Just stumbled across this via #1932.
See #1932 (comment) for my comment on it.

These tests cannot be easily post mortem debugged (pytest --pdb), as unittests tearDown and cleanup (which close the GUI) are called on an exception before the post mortem debugger jumps into place. Therefore, the GUI is closed and no direct interaction is possible any more.

Why is pdb not invoked before the teardown, if the error is before it?
I think that's the real issue, and just skipping teardown for all tests in general when --pdb is used (which some might even use as a default..!) is rather bad.

[blueyed]

Ok, current code looks a bit differrent:
https://github.com/blueyed/pytest/blob/1beba4e69517e69c872fc4a04bed81f9bae1610a/src/_pytest/unittest.py#L219-L226

It might be fine then - let's followup on #1932 / close it if fixed.

[blueyed]

#5996 fixes this.