Allow traceback objects to be instantiated/mutated/annotated

[njsmith]

BPO	30579
Nosy	@brettcannon, @jcea, @ncoghlan, @njsmith, @ericsnowcurrently, @1st1, @miss-islington
PRs	bpo-30579: Allow TracebackType creation and tb_next mutation from Python #4793 bpo-30579: Docs for dynamic traceback creation #5653 [3.7] bpo-30579: Docs for dynamic traceback creation (GH-5653) #5655

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-01-07.13:31:58.450>
created_at = <Date 2017-06-06.03:41:53.475>
labels = ['interpreter-core', 'type-feature', '3.7']
title = 'Allow traceback objects to be instantiated/mutated/annotated'
updated_at = <Date 2018-06-09.13:58:32.107>
user = 'https://github.com/njsmith'

bugs.python.org fields:

activity = <Date 2018-06-09.13:58:32.107>
actor = 'jcea'
assignee = 'none'
closed = True
closed_date = <Date 2018-01-07.13:31:58.450>
closer = 'ncoghlan'
components = ['Interpreter Core']
creation = <Date 2017-06-06.03:41:53.475>
creator = 'njs'
dependencies = []
files = []
hgrepos = []
issue_num = 30579
keywords = ['patch']
message_count = 11.0
messages = ['295235', '295236', '295237', '295238', '295248', '308023', '309532', '309615', '309616', '312107', '312109']
nosy_count = 7.0
nosy_names = ['brett.cannon', 'jcea', 'ncoghlan', 'njs', 'eric.snow', 'yselivanov', 'miss-islington']
pr_nums = ['4793', '5653', '5655']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue30579'
versions = ['Python 3.7']

[njsmith]

Currently, traceback objects don't expose any public constructor, are immutable, and don't have a __dict__ or allow subclassing, which makes it impossible to add extra annotations them.

It would be nice if these limitations were lifted, because there are rare but important cases where code needs to manipulate tracebacks directly, and currently these have to use awful stuff like ctypes. For example:

Jinja2:
https://github.com/pallets/jinja/blob/bec0065c4e7e89e7d893eda7840ba0219824b23c/jinja2/debug.py#L298-L372

Trio:
https://github.com/python-trio/trio/blob/496493afecc22d7d1a17175b6a2748a9c3510066/trio/_core/_multierror.py#L233-L318

(Notice that on PyPy there are no ctypes, but instead they have special extension that's supported for this case only to keep jinja2 working.)

For the above cases, what's needed is the ability to instantiate and assign to the fields of traceback objects.

In addition, in trio I'd like to be able to annotate traceback objects so that the traceback printing machinery can do things like hide "internal" tracebacks, or highlight places where exceptions jumped between tasks. This would be much easier if there were some way to attach data to tracebacks. Probably it doesn't make sense for traceback objects to have a __dict__ by default for speed/memory reasons, but we could add a dedicated metadata slot that is normally empty but can have arbitrary data assigned, or if they allowed subclassing then I could add a __dict__ in a subclass

I'm CC'ing the "import machinery" interest list, because my understanding is that with the new import system there's a similar desire to hide "internal" traceback frames, and while the features requested in this bug report won't solve that problem directly they might (are intended to) provide some useful machinery for it. Feel free to un-CC if I'm wrong :Jinja2:
https://github.com/pallets/jinja/blob/bec0065c4e7e89e7d893eda7840ba0219824b23c/jinja2/debug.py#L298-L372
-).

I think that this should be very straightforward to implement: it's just that no-one ever implemented setters etc. for tracebacks.

[njsmith]

Uh, please ignore the random second paste of the jinja2 URL in the middle of the second to last paragraph.

[ncoghlan]

Rather than allowing this on tracebacks directly, I'd prefer to build on the "TracebackException" work and allow traceback summaries in all the places where we currently require real tracebacks (including exception __traceback__ attributes): https://docs.python.org/3/library/traceback.html#tracebackexception-objects

The essential requirement here would be to abstract out a "traceback protocol" from the current concrete traceback objects, and adapt affected parts of the interpreter (such as the traceback display on shutdown) to use that abstract protocol when dealing with a non-native traceback object.

[njsmith]

My understanding is that the major difference between a real traceback object and a TracebackException object is that the latter is specialized for printing, so it can be lighter weight (no pinning of frame objects in memory), but loses some utility (can't do post-mortem debugging).

If that's right, then that's definitely not a solution, because trio and jinja2 and import errors all need to support post-mortem debugging.

I'm not against the idea of defining a traceback protocol, but it seems like a lot of work when the existing traceback objects are already perfectly good container objects that are just missing a few simple features.

[ncoghlan]

Aye, if you need full post-mortem debugging support, then updating the public API of the actual traceback objects based on the current ctypes-based workaround does seem like a better path forward.

TracebackException and friends would more come into play for the "make traceback printing easier to customise" part of your presentation.

[njsmith]

PR 4793 provides everything that Jinja2 and Trio actually need right now. It doesn't provide any way to annotate tracebacks with extra data, but I'm not entirely sure if that's necessary or what it would look like, so I figured I'd at least get this part in for 3.7.

[njsmith]

Ping -- anyone up for reviewing PR 4793?

#4793

It's pretty straightforward, and I figure better to ping now and beat the end-of-month rush :-)

[ncoghlan]

New changeset e46a8af by Nick Coghlan (Nathaniel J. Smith) in branch 'master':
bpo-30579: Allow TracebackType creation and tb_next mutation from Python (GH-4793)
e46a8af

[ncoghlan]

Thanks for the patch!

[ncoghlan]

New changeset aec7532 by Nick Coghlan in branch 'master':
bpo-30579: Docs for dynamic traceback creation (GH-5653)
aec7532

[miss-islington]

New changeset 53374cc by Miss Islington (bot) in branch '3.7':
bpo-30579: Docs for dynamic traceback creation (GH-5653)
53374cc