bpo-29587: Enable implicit exception chaining with gen.throw() #19823

cjerdonek · 2020-04-30T21:43:45Z

This is a new version of PR #19811 to sort out the buildbot failures.

It enables implicit exception chaining when calling generator.throw(exc) by setting exc.__context__.

https://bugs.python.org/issue29587

Before this commit, if an exception was active inside a generator when calling gen.throw(), then that exception was lost (i.e. there was no implicit exception chaining). This commit fixes that.

Objects/genobject.c

vstinner

test_unittest still crash on this PR. I tested on Linux (Fedora 32).

$ make clean
$ make
$ ./python -m test -v test_unittest 
(...)
test_patch_nested_autospec_repr (unittest.test.testmock.testpatch.PatchTest) ... ok
test_patch_object_keyword_args (unittest.test.testmock.testpatch.PatchTest) ... ok
test_patch_object_with_spec_as_boolean (unittest.test.testmock.testpatch.PatchTest) ... ok
test_patch_orderdict (unittest.test.testmock.testpatch.PatchTest) ... ok
test_patch_propagates_exc_on_exit (unittest.test.testmock.testpatch.PatchTest) ... Fatal Python error: Segmentation fault

Current thread 0x00007f886bd1c740 (most recent call first):
  File "/home/vstinner/python/master/Lib/unittest/mock.py", line 1524 in __exit__
  File "/home/vstinner/python/master/Lib/unittest/test/testmock/testpatch.py", line 1673 in __exit__
  File "/home/vstinner/python/master/Lib/contextlib.py", line 498 in __exit__
  File "/home/vstinner/python/master/Lib/unittest/mock.py", line 1323 in decoration_helper
  File "/home/vstinner/python/master/Lib/contextlib.py", line 135 in __exit__
  File "/home/vstinner/python/master/Lib/unittest/mock.py", line 1337 in patched
  File "/home/vstinner/python/master/Lib/unittest/case.py", line 201 in handle
  File "/home/vstinner/python/master/Lib/unittest/case.py", line 733 in assertRaises
  File "/home/vstinner/python/master/Lib/unittest/test/testmock/testpatch.py", line 1692 in test_patch_propagates_exc_on_exit
(...)

bedevere-bot · 2020-04-30T22:13:26Z

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

cjerdonek · 2020-04-30T22:26:51Z

Is there a way to mark this PR as "don't merge" for now? This is the same PR as before, so it has the same issue. The Azure CI job seems not to be running, which is what exhibited the failure before.

test_unittest still crash on this PR. I tested on Linux (Fedora 32).

Yes, this is the same failure as before. Everything passes locally on my Mac, so it will take me more time to look into this.

vstinner · 2020-04-30T22:51:14Z

Is there a way to mark this PR as "don't merge" for now?

Hum, I wasn't sure, so I did the 3 tricks that I know:

Add "do no merge label"
Add "[WIP]" in the PR title
Convert the PR to a draft

It seems like only the conversion to a draft technically prevents someone to merge the PR my mistake.

cjerdonek · 2020-04-30T22:52:08Z

Looks like that will cover it. ;)

cjerdonek · 2020-05-01T08:42:37Z

Okay, I believe my latest changes fix the buildbot failures from before.

Also, for future reference, below is a script I came up with that isolates the behavior difference between Mac and Fedora, with my prior version of the PR:

import re
import sys


def f():
    exc, exc_value, tb = sys.exc_info()
    print(f'CLEARING FRAME: {tb.tb_frame!r}')
    tb.tb_frame.clear()

def g():
    # Uncommenting the following line caused the tb_frame.clear() line
    # above to exhibit the following platform-specific behavior:
    # 1) On Mac, this is logged to stderr
    #  > TypeError: print_exception(): Exception expected for value,
    #    NoneType found
    # 2) On Fedora 32, the following error happens:
    #  > Fatal Python error: Segmentation fault
    data = re.compile('xxx')

    try:
        yield
    except Exception:
        f()

gen = g()
gen.send(None)
gen.throw(ValueError)

Maybe this suggests an issue elsewhere in Python's code base.

Should this PR also get a What's New?

cjerdonek · 2020-05-01T08:47:01Z

I have made the requested changes; please review again.

bedevere-bot · 2020-05-01T08:47:04Z

Thanks for making the requested changes!

@vstinner: please review the changes made to this pull request.

vstinner

LGTM. "./python -m test -j0 -r" does no longer crash on my Fedora.

I understood that passing NULL value to _PyErr_ChainExceptions() created the bug? Maybe add an assertion to prevent the situation to happen again?

cjerdonek · 2020-05-01T13:33:21Z

Thank you.

Regarding an assertion, the thing is that it looks like _PyErr_ChainExceptions() is designed to accept a NULL value. So maybe in the future it can be made to work. (That's why I added an XXX comment.) So I wouldn't want to add an assertion making it seem like it always has to be like that. It could just something about how things are set up right now that was causing the crash.

Would you still like me to add an assertion and/or perhaps a code comment explaining this possibility?

bedevere-bot · 2020-05-02T01:14:22Z

@cjerdonek: Please replace # with GH- in the commit message next time. Thanks!

This is a follow-up to GH-19823 that removes the check that the exception value isn't NULL, prior to calling _PyErr_ChainExceptions(). This enables implicit exception chaining for gen.throw() in more circumstances. The commit also adds a test that a particular code snippet involving gen.throw() doesn't crash. The test shows why the new `gi_exc_state.exc_type != Py_None` check that was added is necessary. Without the new check, the code snippet (as well as a number of other tests) crashes on certain platforms (e.g. Fedora but not Mac).

cjerdonek · 2020-05-03T07:41:36Z

For future reference, I was able to remove the NULL check in a later PR (#19877): 21893fb

cjerdonek added 2 commits April 30, 2020 14:38

Add a failing test for issue 29587.

e757c76

bpo-29587: set exc.__context__ when calling gen.throw() (pythonGH-19811)

57597f2

Before this commit, if an exception was active inside a generator when calling gen.throw(), then that exception was lost (i.e. there was no implicit exception chaining). This commit fixes that.

cjerdonek requested a review from 1st1 as a code owner April 30, 2020 21:43

the-knights-who-say-ni added the CLA signed label Apr 30, 2020

bedevere-bot added the awaiting core review label Apr 30, 2020

1st1 reviewed Apr 30, 2020

View reviewed changes

Objects/genobject.c Outdated Show resolved Hide resolved

vstinner requested changes Apr 30, 2020

View reviewed changes

bedevere-bot removed the awaiting core review label Apr 30, 2020

bedevere-bot added the awaiting changes label Apr 30, 2020

vstinner added the DO-NOT-MERGE label Apr 30, 2020

vstinner changed the title ~~bpo-29587: Enable implicit exception chaining with gen.throw()~~ [WIP] bpo-29587: Enable implicit exception chaining with gen.throw() Apr 30, 2020

vstinner marked this pull request as draft April 30, 2020 22:49

cjerdonek added 2 commits May 1, 2020 01:09

Wrap the long line.

c321731

Address the CI failures on Fedora, etc.

1bf3309

cjerdonek removed the DO-NOT-MERGE label May 1, 2020

cjerdonek marked this pull request as ready for review May 1, 2020 08:43

cjerdonek changed the title ~~[WIP] bpo-29587: Enable implicit exception chaining with gen.throw()~~ bpo-29587: Enable implicit exception chaining with gen.throw() May 1, 2020

bedevere-bot added awaiting change review and removed awaiting changes labels May 1, 2020

bedevere-bot requested a review from vstinner May 1, 2020 08:47

vstinner approved these changes May 1, 2020

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting change review labels May 1, 2020

cjerdonek merged commit 0204726 into python:master May 2, 2020

bedevere-bot removed the awaiting merge label May 2, 2020

cjerdonek deleted the fix-issue-29587-2 branch May 2, 2020 01:16

This was referenced May 2, 2020

bpo-29587: Enable exception chaining for gen.throw() with "yield from" #19858

Merged

bpo-29587: Add another test for the gen.throw() exception chain fix #19859

Merged

bpo-29587: Remove the exc_value NULL check in _gen_throw() #19877

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-29587: Enable implicit exception chaining with gen.throw() #19823

bpo-29587: Enable implicit exception chaining with gen.throw() #19823

cjerdonek commented Apr 30, 2020 •

edited by bedevere-bot

Loading

vstinner left a comment

bedevere-bot commented Apr 30, 2020

cjerdonek commented Apr 30, 2020

vstinner commented Apr 30, 2020 •

edited

Loading

cjerdonek commented Apr 30, 2020

cjerdonek commented May 1, 2020 •

edited

Loading

cjerdonek commented May 1, 2020

bedevere-bot commented May 1, 2020

vstinner left a comment

cjerdonek commented May 1, 2020

bedevere-bot commented May 2, 2020

cjerdonek commented May 3, 2020

bpo-29587: Enable implicit exception chaining with gen.throw() #19823

bpo-29587: Enable implicit exception chaining with gen.throw() #19823

Conversation

cjerdonek commented Apr 30, 2020 • edited by bedevere-bot Loading

vstinner left a comment

Choose a reason for hiding this comment

bedevere-bot commented Apr 30, 2020

cjerdonek commented Apr 30, 2020

vstinner commented Apr 30, 2020 • edited Loading

cjerdonek commented Apr 30, 2020

cjerdonek commented May 1, 2020 • edited Loading

cjerdonek commented May 1, 2020

bedevere-bot commented May 1, 2020

vstinner left a comment

Choose a reason for hiding this comment

cjerdonek commented May 1, 2020

bedevere-bot commented May 2, 2020

cjerdonek commented May 3, 2020

cjerdonek commented Apr 30, 2020 •

edited by bedevere-bot

Loading

vstinner commented Apr 30, 2020 •

edited

Loading

cjerdonek commented May 1, 2020 •

edited

Loading