Fix some flaky reporting corner cases

[jobh]

Closes #4183
Closes #4228

Two corner cases in reporting of flaky errors.

One of them is a straightforward missing handling of flakiness at mark_interesting time.

The other deals with a surprising aspect of how a naked exception is wrapped in an ExceptionGroup when caught and reraised by except* — the group's traceback points not to where the naked exception is raised, nor to the location of the except* block, but instead to the caller one frame above the except*:

def wrapped():
    try:
        raise Exception  # innermost frame of EceptionGroup.exceptions[0]
    except* Exception:
        raise

def naked():
    raise Exception  # innermost frame of Exception

if __name__ == "__main__":
    import traceback
    try:
        naked()
    except Exception as e:
        print("naked:", traceback.format_tb(e.__traceback__)[-1])
    try:
        wrapped()  # innermost frame of ExceptionGroup
    except Exception as e:
        print("wrapper:", traceback.format_tb(e.__traceback__)[-1])
        print("wrapped:", traceback.format_tb(e.exceptions[0].__traceback__)[-1])

I'm not sure if this is consistent between python implementations, so the final commit adds a check. Will revert once CI results are in.

[Zac-HD]

Thanks Joachim! This looks great and I'm sure will be increasingly useful over time 😁

[Zac-HD]

What if there are multiple layers, e.g. ExceptionGroup("outer", ExceptionGroup("inner", [ValueError()]))? Should we use while to unwrap them all?

(I'd appreciate a code comment about this so we're less baffled when revisiting in a few years for something even more complicated 😅)

[jobh]

Yep done.

The "missing stack frame" on implicit wrapping of naked exceptions in except* — I won't call it a cpython bug, as it doesn't contradict any documentation I've seen, but IMO it's surprising behaviour. Do you think it would be worth raising a concern upstream @Zac-HD ?

[Zac-HD]

Yeah, I think it's worth opening an issue to say something like "we were surprised by this behavior in $PR, and think that $proposal would be less surprising and more useful. If you agree, up to you whether this treated as a bugfix, or a minor change for 3.14"

[jobh]

Too late for a code comment in this PR, but FYI @Zac-HD : Cpython will have this fixed in 3.12+, so the workaround can be removed once 3.11 is EOL.

[tybug]

Nice! I've stuck a note to that effect onto #4252.

[tybug]

Test failures are at least in part my fault - I'll look into this and come back with a fix.

FAILED tests/conjecture/test_shrinker.py::test_redistribute_numeric_pairs - KeyError: 'stopped-because'

[tybug]

#4240 should address failures, and in the meantime we can retry jobs that fail because of it.

[Zac-HD]

Thanks again, all!

[carterbox]

Thanks, @jobh for patching this quickly!