Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multiple PyFrameObjects can be created for a single _PyInterpreterFrame #97002

Closed
davfsa opened this issue Sep 21, 2022 · 37 comments
Closed

Multiple PyFrameObjects can be created for a single _PyInterpreterFrame #97002

davfsa opened this issue Sep 21, 2022 · 37 comments
Assignees
@brandtbucher
Labels
3.11 only security fixes 3.12 bugs and security fixes release-blocker sprint type-crash A hard crash of the interpreter, possibly with a core dump

Comments

@davfsa
Copy link

davfsa commented Sep 21, 2022

Crash report

I am not really sure under what circumstances this can occur, as I have not been able to isolate the specific piece of code that triggers this. Thankfully, when trying to re-create the segfault to capture a core dump locally, I have found that I instead got a full traceback, which you can find bellow

Error messages

Segfault 
============================= test session starts ==============================
platform linux -- Python 3.11.0rc2, pytest-7.1.3, pluggy-1.0.0
Using --randomly-seed=3689645484
rootdir: /home/runner/work/hikari/hikari, configfile: pyproject.toml
plugins: cov-3.0.0, randomly-3.12.0, asyncio-0.19.0
asyncio: mode=Mode.STRICT
collected 3244 items

tests/hikari/internal/test_aio.py ....................................   [  1%]
tests/hikari/test_templates.py .                                         [  1%]
tests/hikari/impl/test_event_manager.py Fatal Python error: Segmentation fault

Current thread 0x00007f5aff3b8740 (most recent call first):
  File "/opt/hostedtoolcache/Python/3.11.0-rc.2/x64/lib/python3.11/traceback.py", line 366 in _get_code_position
  File "/opt/hostedtoolcache/Python/3.11.0-rc.2/x64/lib/python3.11/traceback.py", line 353 in _walk_tb_with_full_positions
  File "/opt/hostedtoolcache/Python/3.11.0-rc.2/x64/lib/python3.11/traceback.py", line 416 in _extract_from_extended_frame_gen
  File "/opt/hostedtoolcache/Python/3.11.0-rc.2/x64/lib/python3.11/traceback.py", line 688 in __init__
  File "/opt/hostedtoolcache/Python/3.11.0-rc.2/x64/lib/python3.11/traceback.py", line 139 in format_exception
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/_pytest/unraisableexception.py", line 72 in unraisable_exception_runtest_hook
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/_pytest/unraisableexception.py", line 83 in pytest_runtest_setup
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/pluggy/_callers.py", line 55 in _multicall
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/_pytest/runner.py", line 259 in <lambda>
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/_pytest/runner.py", line 338 in from_call
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/_pytest/runner.py", line 258 in call_runtest_hook
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/_pytest/runner.py", line 219 in call_and_report
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/_pytest/runner.py", line 124 in runtestprotocol
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/_pytest/runner.py", line 111 in pytest_runtest_protocol
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/_pytest/main.py", line 347 in pytest_runtestloop
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/_pytest/main.py", line 322 in _main
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/_pytest/main.py", line 268 in wrap_session
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/_pytest/main.py", line 315 in pytest_cmdline_main
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/_pytest/config/__init__.py", line 164 in main
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/_pytest/config/__init__.py", line 187 in console_main
  File "/home/runner/work/hikari/hikari/.nox/pytest/lib/python3.11/site-packages/pytest/__main__.py", line 5 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 198 in _run_module_as_main

Extension modules: multidict._multidict, yarl._quoting_c, aiohttp._helpers, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket, frozenlist._frozenlist (total: 7)
Traceback 
exc = <class 'ResourceWarning'>, value = ResourceWarning('unclosed <socket.socket fd=16, family=1, type=1, proto=0>'), tb = <traceback object at 0x7f1e410ba440>, limit = None, chain = True

    def format_exception(exc, /, value=_sentinel, tb=_sentinel, limit=None, \
                         chain=True):
        """Format a stack trace and the exception information.
    
        The arguments have the same meaning as the corresponding arguments
        to print_exception().  The return value is a list of strings, each
        ending in a newline and some containing internal newlines.  When
        these lines are concatenated and printed, exactly the same text is
        printed as does print_exception().
        """
        value, tb = _parse_value_tb(exc, value, tb)
>       te = TracebackException(type(value), value, tb, limit=limit, compact=True)

chain      = True
exc        = <class 'ResourceWarning'>
limit      = None
tb         = <traceback object at 0x7f1e410ba440>
value      = ResourceWarning('unclosed <socket.socket fd=16, family=1, type=1, proto=0>')

/usr/lib/python3.11/traceback.py:139: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python3.11/traceback.py:688: in __init__
    self.stack = StackSummary._extract_from_extended_frame_gen(
        _seen      = {139767898821088}
        capture_locals = False
        compact    = True
        exc_traceback = <traceback object at 0x7f1e410ba440>
        exc_type   = <class 'ResourceWarning'>
        exc_value  = ResourceWarning('unclosed <socket.socket fd=16, family=1, type=1, proto=0>')
        is_recursive_call = False
        limit      = None
        lookup_lines = True
        max_group_depth = 10
        max_group_width = 15
        self       = <traceback.TracebackException object at 0x7f1e3fddb410>
/usr/lib/python3.11/traceback.py:416: in _extract_from_extended_frame_gen
    for f, (lineno, end_lineno, colno, end_colno) in frame_gen:
        capture_locals = False
        fnames     = set()
        frame_gen  = <generator object _walk_tb_with_full_positions at 0x7f1e3f1ecba0>
        klass      = <class 'traceback.StackSummary'>
        limit      = None
        lookup_lines = True
        result     = []
/usr/lib/python3.11/traceback.py:353: in _walk_tb_with_full_positions
    positions = _get_code_position(tb.tb_frame.f_code, tb.tb_lasti)
        tb         = <traceback object at 0x7f1e410ba440>
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

code = <cell at 0x7f1e45de9f60: str object at 0x7f1e45bd80f0>, instruction_index = 0

    def _get_code_position(code, instruction_index):
        if instruction_index < 0:
            return (None, None, None, None)
>       positions_gen = code.co_positions()
E       AttributeError: 'cell' object has no attribute 'co_positions'

code       = <cell at 0x7f1e45de9f60: str object at 0x7f1e45bd80f0>
instruction_index = 0

/usr/lib/python3.11/traceback.py:366: AttributeError

-------- coverage: platform linux, python 3.11.0-candidate-2 ---------

Your environment

  • CPython versions tested on: 3.11.0-rc.2
  • Operating system and architecture: Linux x86_64
@davfsa davfsa added the type-crash A hard crash of the interpreter, possibly with a core dump label Sep 21, 2022
davfsa added a commit to davfsa/hikari that referenced this issue Sep 21, 2022
@davfsa
Temporarily disable testing in 3.11-dev due to CPython crash
5dfb900 
ref: python/cpython#97002
@davfsa davfsa mentioned this issue Sep 21, 2022
Merged
2 tasks
@brandtbucher
Copy link
Member

@pablogsal, this looks a lot like #95818. Possibly related to #96975?

I'll dig into it further.

@brandtbucher brandtbucher added release-blocker 3.11 only security fixes labels Sep 21, 2022
@brandtbucher
Copy link
Member

@davfsa, does this crash pretty consistently? I'm unable to reproduce on main, 3.11, or v3.11.0rc2... Pytest just runs the test suite and passes.

Do you have concrete (ideally, minimal) steps that I can follow to reproduce it? Or maybe a core dump?

@davfsa
Copy link
Author

davfsa commented Sep 22, 2022 •
edited
Loading

@davfsa, does this crash pretty consistently? I'm unable to reproduce on main, 3.11, or v3.11.0rc2... Pytest just runs the test suite and passes.

Do you have concrete (ideally, minimal) steps that I can follow to reproduce it? Or maybe a core dump?

I'll check when I get home, but before opening this issue I ran our test suite several times and was quite consistent. The segfault happens only on our CI, so might be a bit hard to get a core dump from there. I'll try to get one locally.

So far the underlying issue I found happens when aiohttp leaves an open websocket, leading to pytest to wrap it and raise a warning, but we have it setup to error on any warnings.

I'll try go get a minimally repo as soon as I can, but the issue can be found when running our test suite (https://github.com/hikari-py/hikari) using nox -s pytest.

@brandtbucher
Copy link
Member

I am able to reproduce this intermittently now, partially minimized. What I'm doing is running:

$ pytest -v tests/hikari/impl/test_rest_bot.py --cov

Sometimes we get those attribute errors in the tracebacks where frames/code/etc. are being replaced with seemingly random objects, and sometimes it just segfaults. So this does look like a use-after-free somewhere, similar to #95818.

Worth noting is that this only seems to happen when --cov is passed. So C tracing seems like a possible culprit.

@brandtbucher
Copy link
Member

brandtbucher commented Sep 22, 2022 •
edited
Loading

With a debug build, I'm (sometimes) able to observe the following assertion failure during tests/hikari/impl/test_rest_bot.py::TestRESTBot::test_get_listener:

Objects/frameobject.c:1304: PyFrame_GetCode: Assertion `code != NULL' failed.
Fatal Python error: Aborted

Current thread 0x00007f86102d0280 (most recent call first):
  File "/home/brandtbucher/cpython/Lib/traceback.py", line 353 in _walk_tb_with_full_positions
  File "/home/brandtbucher/cpython/Lib/traceback.py", line 416 in _extract_from_extended_frame_gen
  File "/home/brandtbucher/cpython/Lib/traceback.py", line 688 in __init__
  File "/home/brandtbucher/cpython/Lib/traceback.py", line 139 in format_exception
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/unraisableexception.py", line 72 in unraisable_exception_runtest_hook
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/unraisableexception.py", line 83 in pytest_runtest_setup
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_callers.py", line 55 in _multicall
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/runner.py", line 259 in <lambda>
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/runner.py", line 338 in from_call
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/runner.py", line 258 in call_runtest_hook
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/runner.py", line 219 in call_and_report
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/runner.py", line 124 in runtestprotocol
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/runner.py", line 111 in pytest_runtest_protocol
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/main.py", line 347 in pytest_runtestloop
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/main.py", line 322 in _main
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/main.py", line 268 in wrap_session
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/main.py", line 315 in pytest_cmdline_main
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/config/__init__.py", line 164 in main
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/config/__init__.py", line 187 in console_main
  File "/home/brandtbucher/env311/bin/pytest", line 8 in <module>

Full dump:

#0  raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x000055878e6ede65 in faulthandler_fatal_error (signum=6) at ./Modules/faulthandler.c:373
#2  <signal handler called>
#3  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#4  0x00007fdbd477a859 in __GI_abort () at abort.c:79
#5  0x00007fdbd477a729 in __assert_fail_base (fmt=0x7fdbd4910588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x55878e7bb321 "code != NULL", file=0x55878e7a2af4 "Objects/frameobject.c", line=1304, function=<optimized out>) at assert.c:92
#6  0x00007fdbd478bf36 in __GI___assert_fail (assertion=assertion@entry=0x55878e7bb321 "code != NULL", file=file@entry=0x55878e7a2af4 "Objects/frameobject.c", line=line@entry=1304, function=function@entry=0x55878e7a3640 <__PRETTY_FUNCTION__.18739> "PyFrame_GetCode") at assert.c:101
#7  0x000055878e55e344 in PyFrame_GetCode (Python Exception <class '__main__.NullPyObjectPtr'> <__main__.PyCodeObjectPtr object at 0x7f09f19d6df0>: frame=frame@entry=) at Objects/frameobject.c:1306
#8  0x000055878e55e37a in frame_getcode (Python Exception <class '__main__.NullPyObjectPtr'> <__main__.PyCodeObjectPtr object at 0x7f09f19d6670>: f=, closure=<optimized out>) at Objects/frameobject.c:97
#9  0x000055878e54223b in getset_get (descr=descr@entry=0x7fdbd4385850, Python Exception <class 'gdb.error'> There is no member named f_frame.: obj=obj@entry=, type=<optimized out>) at Objects/descrobject.c:202
#10 0x000055878e599bf9 in _PyObject_GenericGetAttrWithDict (Python Exception <class 'gdb.error'> There is no member named f_frame.: obj=, name='f_code', dict=dict@entry=0x0, suppress=suppress@entry=0) at ./Include/object.h:133
#11 0x000055878e59a520 in PyObject_GenericGetAttr (obj=<optimized out>, name=<optimized out>) at Objects/object.c:1367
#12 0x000055878e5998b2 in PyObject_GetAttr (Python Exception <class 'gdb.error'> There is no member named f_frame.: v=v@entry=, name='f_code') at Objects/object.c:916
#13 0x000055878e66121b in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=frame@entry=0x7fdbd0a3d2a0, throwflag=throwflag@entry=0) at Python/ceval.c:3466
#14 0x000055878e553e55 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd0a3d2a0, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#15 gen_send_ex2 (gen=0x7fdbd0a3d250, arg=arg@entry=0x0, presult=presult@entry=0x7ffdc69a9b10, exc=exc@entry=0, closing=closing@entry=0) at Objects/genobject.c:219
#16 0x000055878e55491f in gen_iternext (gen=<optimized out>) at Objects/genobject.c:594
#17 0x000055878e665e67 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d6310, frame@entry=0x7fdbd42d61b0, throwflag=throwflag@entry=0) at ./Include/object.h:133
#18 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d61b0, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#19 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#20 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393
#21 0x000055878e535a72 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd406f490>, args=args@entry=0x7ffdc69a9d30, nargsf=nargsf@entry=4, kwargs=kwargs@entry={'limit': None, 'compact': True}) at Objects/call.c:152
#22 0x000055878e535c31 in _PyObject_Call_Prepend (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd406f490>, obj=obj@entry=<TracebackException(max_group_width=15, max_group_depth=10) at remote 0x7fdbd0aaca60>, args=args@entry=(<type at remote 0x55878e8e23e0>, ResourceWarning('unclosed <socket.socket fd=13, family=1, type=1, proto=0>',), <traceback at remote 0x7fdbd071fb60>), kwargs=kwargs@entry={'limit': None, 'compact': True}) at Objects/call.c:482
#23 0x000055878e5bcda1 in slot_tp_init (self=<TracebackException(max_group_width=15, max_group_depth=10) at remote 0x7fdbd0aaca60>, args=(<type at remote 0x55878e8e23e0>, ResourceWarning('unclosed <socket.socket fd=13, family=1, type=1, proto=0>',), <traceback at remote 0x7fdbd071fb60>), kwds={'limit': None, 'compact': True}) at Objects/typeobject.c:7861
#24 0x000055878e5b83e1 in type_call (type=0x55878f938370, args=(<type at remote 0x55878e8e23e0>, ResourceWarning('unclosed <socket.socket fd=13, family=1, type=1, proto=0>',), <traceback at remote 0x7fdbd071fb60>), kwds={'limit': None, 'compact': True}) at Objects/typeobject.c:1112
#25 0x000055878e535764 in _PyObject_MakeTpCall (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<type at remote 0x55878f938370>, args=args@entry=0x7fdbd42d6188, nargs=<optimized out>, keywords=keywords@entry=('limit', 'compact')) at Objects/call.c:214
#26 0x000055878e5361fb in _PyObject_VectorcallTstate (kwnames=('limit', 'compact'), nargsf=9223372036854775811, args=0x7fdbd42d6188, callable=<type at remote 0x55878f938370>, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:90
#27 PyObject_Vectorcall (callable=callable@entry=<type at remote 0x55878f938370>, args=args@entry=0x7fdbd42d6188, nargsf=<optimized out>, nargsf@entry=9223372036854775811, kwnames=kwnames@entry=('limit', 'compact')) at Objects/call.c:299
#28 0x000055878e6576a2 in trace_call_function (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<type at remote 0x55878f938370>, args=args@entry=0x7fdbd42d6188, nargs=<optimized out>, kwnames=('limit', 'compact')) at Python/ceval.c:7305
#29 0x000055878e667ca9 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d6100, frame@entry=0x7fdbd0967c40, throwflag=throwflag@entry=0) at Python/ceval.c:4767
#30 0x000055878e553e55 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd0967c40, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#31 gen_send_ex2 (gen=gen@entry=0x7fdbd0967bf0, arg=<_Result(_result=[], _excinfo=None) at remote 0x7fdbd0762450>, presult=presult@entry=0x7ffdc69aa090, exc=exc@entry=0, closing=closing@entry=0) at Objects/genobject.c:219
#32 0x000055878e554236 in gen_send_ex (gen=0x7fdbd0967bf0, arg=<optimized out>, exc=exc@entry=0, closing=closing@entry=0) at Objects/genobject.c:287
#33 0x000055878e5548b8 in gen_send (gen=<optimized out>, arg=<optimized out>) at Objects/genobject.c:310
#34 0x000055878e545401 in method_vectorcall_O (func=<method_descriptor at remote 0x7fdbd4387110>, args=0x7ffdc69aa230, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/descrobject.c:481
#35 0x000055878e536905 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7ffdc69aa230, callable=<method_descriptor at remote 0x7fdbd4387110>, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:92
#36 PyObject_VectorcallMethod (name=<optimized out>, args=args@entry=0x7ffdc69aa230, nargsf=2, nargsf@entry=9223372036854775810, kwnames=kwnames@entry=0x0) at Objects/call.c:855
#37 0x000055878e65ca0a in PyObject_CallMethodOneArg (arg=<_Result(_result=[], _excinfo=None) at remote 0x7fdbd0762450>, name=<optimized out>, self=<generator at remote 0x7fdbd0967bf0>) at ./Include/cpython/abstract.h:116
#38 _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=frame@entry=0x7fdbd0b4f710, throwflag=throwflag@entry=0) at Python/ceval.c:2598
#39 0x000055878e553e55 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd0b4f710, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#40 gen_send_ex2 (gen=gen@entry=0x7fdbd0b4f6c0, arg=<_Result(_result=[], _excinfo=None) at remote 0x7fdbd0762450>, presult=presult@entry=0x7ffdc69aa2f0, exc=exc@entry=0, closing=closing@entry=0) at Objects/genobject.c:219
#41 0x000055878e554236 in gen_send_ex (gen=0x7fdbd0b4f6c0, arg=<optimized out>, exc=exc@entry=0, closing=closing@entry=0) at Objects/genobject.c:287
#42 0x000055878e5548b8 in gen_send (gen=<optimized out>, arg=<optimized out>) at Objects/genobject.c:310
#43 0x000055878e593230 in cfunction_vectorcall_O (func=<built-in method send of generator object at remote 0x7fdbd0b4f6c0>, args=0x7fdbd42d60e0, nargsf=<optimized out>, kwnames=<optimized out>) at ./Include/cpython/methodobject.h:52
#44 0x000055878e53610a in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=1, args=0x7fdbd42d60e0, callable=<built-in method send of generator object at remote 0x7fdbd0b4f6c0>, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:92
#45 PyObject_Vectorcall (callable=callable@entry=<built-in method send of generator object at remote 0x7fdbd0b4f6c0>, args=args@entry=0x7fdbd42d60e0, nargsf=nargsf@entry=1, kwnames=kwnames@entry=0x0) at Objects/call.c:299
#46 0x000055878e657908 in trace_call_function (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=<built-in method send of generator object at remote 0x7fdbd0b4f6c0>, func@entry=<method_descriptor at remote 0x7fdbd4387110>, args=args@entry=0x7fdbd42d60d8, nargs=2, kwnames=0x0) at Python/ceval.c:7299
#47 0x000055878e667ca9 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d6010, frame@entry=0x7fdbd42d5ec8, throwflag=throwflag@entry=0) at Python/ceval.c:4767
#48 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d5ec8, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#49 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#50 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393
#51 0x000055878e535a72 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3e18100>, args=args@entry=0x7ffdc69aa5d0, nargsf=nargsf@entry=1, kwargs=kwargs@entry={'item': <Function at remote 0x7fdbd0a83f60>}) at Objects/call.c:152
#52 0x000055878e535c31 in _PyObject_Call_Prepend (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3e18100>, obj=obj@entry=<_HookCaller(name='pytest_runtest_setup', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd3486fc0>, argnames=(), kwargnames=(...), plugin=<module at remote 0x7fdbd32505f0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='threadexception', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd348fb70>, <HookImpl(function=<method at remote 0x7fdbd2769370>, argnames=('item',), kwargnames=(...), plugin=<CaptureManager(_method='fd', _global_capturing=<MultiCapture(in_=<FDCapture(targetfd=0, targetfd_invalid=None, targetfd_save=3, tmpfile=<_io.TextIOWrapper at remote 0x7fdbd2657110>, syscapture=<SysCapture(_old=<_io.TextIOWrapper at remote 0x7fdbd41de120>, name='stdin', tmpfile=<DontReadFromInput() at remote 0x7fdbd26d4830>, _state='started') at remote 0x7fdbd26d4240>, _state='started') at remote 0x7fdbd3d1aef0>, out=<FDCapture(targetfd=1, targetfd_invalid=None, targetfd_save=5, tmpfile=<Enco...(truncated), args=args@entry=(), kwargs=kwargs@entry={'item': <Function at remote 0x7fdbd0a83f60>}) at Objects/call.c:482
#53 0x000055878e5bcf65 in slot_tp_call (self=<_HookCaller(name='pytest_runtest_setup', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd3486fc0>, argnames=(), kwargnames=(...), plugin=<module at remote 0x7fdbd32505f0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='threadexception', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd348fb70>, <HookImpl(function=<method at remote 0x7fdbd2769370>, argnames=('item',), kwargnames=(...), plugin=<CaptureManager(_method='fd', _global_capturing=<MultiCapture(in_=<FDCapture(targetfd=0, targetfd_invalid=None, targetfd_save=3, tmpfile=<_io.TextIOWrapper at remote 0x7fdbd2657110>, syscapture=<SysCapture(_old=<_io.TextIOWrapper at remote 0x7fdbd41de120>, name='stdin', tmpfile=<DontReadFromInput() at remote 0x7fdbd26d4830>, _state='started') at remote 0x7fdbd26d4240>, _state='started') at remote 0x7fdbd3d1aef0>, out=<FDCapture(targetfd=1, targetfd_invalid=None, targetfd_save=5, tmpfile=<Enco...(truncated), args=(), kwds={'item': <Function at remote 0x7fdbd0a83f60>}) at Objects/typeobject.c:7630
#54 0x000055878e5352c9 in _PyObject_Call (tstate=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<_HookCaller(name='pytest_runtest_setup', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd3486fc0>, argnames=(), kwargnames=(...), plugin=<module at remote 0x7fdbd32505f0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='threadexception', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd348fb70>, <HookImpl(function=<method at remote 0x7fdbd2769370>, argnames=('item',), kwargnames=(...), plugin=<CaptureManager(_method='fd', _global_capturing=<MultiCapture(in_=<FDCapture(targetfd=0, targetfd_invalid=None, targetfd_save=3, tmpfile=<_io.TextIOWrapper at remote 0x7fdbd2657110>, syscapture=<SysCapture(_old=<_io.TextIOWrapper at remote 0x7fdbd41de120>, name='stdin', tmpfile=<DontReadFromInput() at remote 0x7fdbd26d4830>, _state='started') at remote 0x7fdbd26d4240>, _state='started') at remote 0x7fdbd3d1aef0>, out=<FDCapture(targetfd=1, targetfd_invalid=None, targetfd_save=5, tmpfile=<Enco...(truncated), args=args@entry=(), kwargs=kwargs@entry={'item': <Function at remote 0x7fdbd0a83f60>}) at Objects/call.c:343
#55 0x000055878e5353f2 in PyObject_Call (callable=callable@entry=<_HookCaller(name='pytest_runtest_setup', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd3486fc0>, argnames=(), kwargnames=(...), plugin=<module at remote 0x7fdbd32505f0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='threadexception', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd348fb70>, <HookImpl(function=<method at remote 0x7fdbd2769370>, argnames=('item',), kwargnames=(...), plugin=<CaptureManager(_method='fd', _global_capturing=<MultiCapture(in_=<FDCapture(targetfd=0, targetfd_invalid=None, targetfd_save=3, tmpfile=<_io.TextIOWrapper at remote 0x7fdbd2657110>, syscapture=<SysCapture(_old=<_io.TextIOWrapper at remote 0x7fdbd41de120>, name='stdin', tmpfile=<DontReadFromInput() at remote 0x7fdbd26d4830>, _state='started') at remote 0x7fdbd26d4240>, _state='started') at remote 0x7fdbd3d1aef0>, out=<FDCapture(targetfd=1, targetfd_invalid=None, targetfd_save=5, tmpfile=<Enco...(truncated), args=args@entry=(), kwargs=kwargs@entry={'item': <Function at remote 0x7fdbd0a83f60>}) at Objects/call.c:355
#56 0x000055878e65797c in do_call_core (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<_HookCaller(name='pytest_runtest_setup', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd3486fc0>, argnames=(), kwargnames=(...), plugin=<module at remote 0x7fdbd32505f0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='threadexception', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd348fb70>, <HookImpl(function=<method at remote 0x7fdbd2769370>, argnames=('item',), kwargnames=(...), plugin=<CaptureManager(_method='fd', _global_capturing=<MultiCapture(in_=<FDCapture(targetfd=0, targetfd_invalid=None, targetfd_save=3, tmpfile=<_io.TextIOWrapper at remote 0x7fdbd2657110>, syscapture=<SysCapture(_old=<_io.TextIOWrapper at remote 0x7fdbd41de120>, name='stdin', tmpfile=<DontReadFromInput() at remote 0x7fdbd26d4830>, _state='started') at remote 0x7fdbd26d4240>, _state='started') at remote 0x7fdbd3d1aef0>, out=<FDCapture(targetfd=1, targetfd_invalid=None, targetfd_save=5, tmpfile=<Enco...(truncated), callargs=callargs@entry=(), kwdict=kwdict@entry={'item': <Function at remote 0x7fdbd0a83f60>}, use_tracing=255) at Python/ceval.c:7347
#57 0x000055878e66b6f1 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d5e40, frame@entry=0x7fdbd42d5cc0, throwflag=throwflag@entry=0) at Python/ceval.c:5377
#58 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d5cc0, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#59 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#60 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393
#61 0x000055878e534f24 in _PyVectorcall_Call (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=0x55878e53541c <_PyFunction_Vectorcall>, callable=callable@entry=<function at remote 0x7fdbd3635ff0>, tuple=tuple@entry=(<Function at remote 0x7fdbd0a83f60>, 'setup'), kwargs=kwargs@entry={}) at ./Include/object.h:772
#62 0x000055878e53538a in _PyObject_Call (tstate=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3635ff0>, args=args@entry=(<Function at remote 0x7fdbd0a83f60>, 'setup'), kwargs=kwargs@entry={}) at Objects/call.c:328
#63 0x000055878e5353f2 in PyObject_Call (callable=callable@entry=<function at remote 0x7fdbd3635ff0>, args=args@entry=(<Function at remote 0x7fdbd0a83f60>, 'setup'), kwargs=kwargs@entry={}) at Objects/call.c:355
#64 0x000055878e65797c in do_call_core (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<function at remote 0x7fdbd3635ff0>, callargs=callargs@entry=(<Function at remote 0x7fdbd0a83f60>, 'setup'), kwdict=kwdict@entry={}, use_tracing=255) at Python/ceval.c:7347
#65 0x000055878e66b6f1 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d5c18, frame@entry=0x7fdbd42d5ae0, throwflag=throwflag@entry=0) at Python/ceval.c:5377
#66 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d5ae0, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#67 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#68 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393
#69 0x000055878e534f24 in _PyVectorcall_Call (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=0x55878e53541c <_PyFunction_Vectorcall>, callable=callable@entry=<function at remote 0x7fdbd3635910>, tuple=tuple@entry=(<Function at remote 0x7fdbd0a83f60>, <Function(keywords=<NodeKeywords at remote 0x7fdbd0a87ac0>, own_markers=[], extra_keyword_matches=set(), stash=<Stash at remote 0x7fdbd0cc0190>, _report_sections=[], user_properties=[], originalname='test_set_listener', _obj=<method at remote 0x7fdbd0a9b770>, _fixtureinfo=<FuncFixtureInfo at remote 0x7fdbd0a9b6b0>, fixturenames=['mock_rest_bot', 'mock_interaction_server', 'mock_rest_client', 'mock_entity_factory', 'mock_http_settings', 'mock_proxy_settings', 'mock_executor'], funcargs={}, _request=<FixtureRequest(_pyfuncitem=<...>, fixturename=None, _scope=<Scope(_value_='function', _name_='Function', __objclass__=<EnumType(_generate_next_value_=<function at remote 0x7fdbd4109230>, __module__='_pytest.scope', __annotations__={'Function': '_ScopeName', 'Class': '_ScopeName', 'Module': '_ScopeName', 'Package': '_ScopeName', 'Session': '_ScopeName'}, __doc__='\n    Represents one of the possible fixture scopes in pytest.\n\n    Scopes are ordered from lower to higher, that i...(truncated), kwargs=kwargs@entry=0x0) at ./Include/object.h:772
#70 0x000055878e53538a in _PyObject_Call (tstate=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3635910>, args=args@entry=(<Function at remote 0x7fdbd0a83f60>, <Function(keywords=<NodeKeywords at remote 0x7fdbd0a87ac0>, own_markers=[], extra_keyword_matches=set(), stash=<Stash at remote 0x7fdbd0cc0190>, _report_sections=[], user_properties=[], originalname='test_set_listener', _obj=<method at remote 0x7fdbd0a9b770>, _fixtureinfo=<FuncFixtureInfo at remote 0x7fdbd0a9b6b0>, fixturenames=['mock_rest_bot', 'mock_interaction_server', 'mock_rest_client', 'mock_entity_factory', 'mock_http_settings', 'mock_proxy_settings', 'mock_executor'], funcargs={}, _request=<FixtureRequest(_pyfuncitem=<...>, fixturename=None, _scope=<Scope(_value_='function', _name_='Function', __objclass__=<EnumType(_generate_next_value_=<function at remote 0x7fdbd4109230>, __module__='_pytest.scope', __annotations__={'Function': '_ScopeName', 'Class': '_ScopeName', 'Module': '_ScopeName', 'Package': '_ScopeName', 'Session': '_ScopeName'}, __doc__='\n    Represents one of the possible fixture scopes in pytest.\n\n    Scopes are ordered from lower to higher, that i...(truncated), kwargs=kwargs@entry=0x0) at Objects/call.c:328
#71 0x000055878e5353f2 in PyObject_Call (callable=callable@entry=<function at remote 0x7fdbd3635910>, args=args@entry=(<Function at remote 0x7fdbd0a83f60>, <Function(keywords=<NodeKeywords at remote 0x7fdbd0a87ac0>, own_markers=[], extra_keyword_matches=set(), stash=<Stash at remote 0x7fdbd0cc0190>, _report_sections=[], user_properties=[], originalname='test_set_listener', _obj=<method at remote 0x7fdbd0a9b770>, _fixtureinfo=<FuncFixtureInfo at remote 0x7fdbd0a9b6b0>, fixturenames=['mock_rest_bot', 'mock_interaction_server', 'mock_rest_client', 'mock_entity_factory', 'mock_http_settings', 'mock_proxy_settings', 'mock_executor'], funcargs={}, _request=<FixtureRequest(_pyfuncitem=<...>, fixturename=None, _scope=<Scope(_value_='function', _name_='Function', __objclass__=<EnumType(_generate_next_value_=<function at remote 0x7fdbd4109230>, __module__='_pytest.scope', __annotations__={'Function': '_ScopeName', 'Class': '_ScopeName', 'Module': '_ScopeName', 'Package': '_ScopeName', 'Session': '_ScopeName'}, __doc__='\n    Represents one of the possible fixture scopes in pytest.\n\n    Scopes are ordered from lower to higher, that i...(truncated), kwargs=kwargs@entry=0x0) at Objects/call.c:355
#72 0x000055878e65797c in do_call_core (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<function at remote 0x7fdbd3635910>, callargs=callargs@entry=(<Function at remote 0x7fdbd0a83f60>, <Function(keywords=<NodeKeywords at remote 0x7fdbd0a87ac0>, own_markers=[], extra_keyword_matches=set(), stash=<Stash at remote 0x7fdbd0cc0190>, _report_sections=[], user_properties=[], originalname='test_set_listener', _obj=<method at remote 0x7fdbd0a9b770>, _fixtureinfo=<FuncFixtureInfo at remote 0x7fdbd0a9b6b0>, fixturenames=['mock_rest_bot', 'mock_interaction_server', 'mock_rest_client', 'mock_entity_factory', 'mock_http_settings', 'mock_proxy_settings', 'mock_executor'], funcargs={}, _request=<FixtureRequest(_pyfuncitem=<...>, fixturename=None, _scope=<Scope(_value_='function', _name_='Function', __objclass__=<EnumType(_generate_next_value_=<function at remote 0x7fdbd4109230>, __module__='_pytest.scope', __annotations__={'Function': '_ScopeName', 'Class': '_ScopeName', 'Module': '_ScopeName', 'Package': '_ScopeName', 'Session': '_ScopeName'}, __doc__='\n    Represents one of the possible fixture scopes in pytest.\n\n    Scopes are ordered from lower to higher, that i...(truncated), kwdict=kwdict@entry=0x0, use_tracing=255) at Python/ceval.c:7347
#73 0x000055878e66b6f1 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d59f0, frame@entry=0x7fdbd42d58a8, throwflag=throwflag@entry=0) at Python/ceval.c:5377
#74 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d58a8, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#75 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#76 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393
#77 0x000055878e535a72 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3e18100>, args=args@entry=0x7ffdc69aadd0, nargsf=nargsf@entry=1, kwargs=kwargs@entry={'item': <Function at remote 0x7fdbd0a83f60>, 'nextitem': <Function(keywords=<NodeKeywords at remote 0x7fdbd0a87ac0>, own_markers=[], extra_keyword_matches=set(), stash=<Stash at remote 0x7fdbd0cc0190>, _report_sections=[], user_properties=[], originalname='test_set_listener', _obj=<method at remote 0x7fdbd0a9b770>, _fixtureinfo=<FuncFixtureInfo at remote 0x7fdbd0a9b6b0>, fixturenames=['mock_rest_bot', 'mock_interaction_server', 'mock_rest_client', 'mock_entity_factory', 'mock_http_settings', 'mock_proxy_settings', 'mock_executor'], funcargs={}, _request=<FixtureRequest(_pyfuncitem=<...>, fixturename=None, _scope=<Scope(_value_='function', _name_='Function', __objclass__=<EnumType(_generate_next_value_=<function at remote 0x7fdbd4109230>, __module__='_pytest.scope', __annotations__={'Function': '_ScopeName', 'Class': '_ScopeName', 'Module': '_ScopeName', 'Package': '_ScopeName', 'Session': '_ScopeName'}, __doc__='\n    Represents one of the possible fixture scopes in pytest.\n\n    Scopes are ordered from low...(truncated)) at Objects/call.c:152
#78 0x000055878e535c31 in _PyObject_Call_Prepend (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3e18100>, obj=obj@entry=<_HookCaller(name='pytest_runtest_protocol', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd34875f0>, argnames=('item',), kwargnames=(), plugin=<module at remote 0x7fdbd3250ef0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='faulthandler', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd3254d80>, <HookImpl(function=<function at remote 0x7fdbd33b7ac0>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd33ee4b0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': False, 'specname': None}, plugin_name='unittest', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd33dfa80>, <HookImpl(function=<function at remote 0x7fdbd3643330>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd4139370>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': True, 'trylast': ...(truncated), args=args@entry=(), kwargs=kwargs@entry={'item': <Function at remote 0x7fdbd0a83f60>, 'nextitem': <Function(keywords=<NodeKeywords at remote 0x7fdbd0a87ac0>, own_markers=[], extra_keyword_matches=set(), stash=<Stash at remote 0x7fdbd0cc0190>, _report_sections=[], user_properties=[], originalname='test_set_listener', _obj=<method at remote 0x7fdbd0a9b770>, _fixtureinfo=<FuncFixtureInfo at remote 0x7fdbd0a9b6b0>, fixturenames=['mock_rest_bot', 'mock_interaction_server', 'mock_rest_client', 'mock_entity_factory', 'mock_http_settings', 'mock_proxy_settings', 'mock_executor'], funcargs={}, _request=<FixtureRequest(_pyfuncitem=<...>, fixturename=None, _scope=<Scope(_value_='function', _name_='Function', __objclass__=<EnumType(_generate_next_value_=<function at remote 0x7fdbd4109230>, __module__='_pytest.scope', __annotations__={'Function': '_ScopeName', 'Class': '_ScopeName', 'Module': '_ScopeName', 'Package': '_ScopeName', 'Session': '_ScopeName'}, __doc__='\n    Represents one of the possible fixture scopes in pytest.\n\n    Scopes are ordered from low...(truncated)) at Objects/call.c:482
#79 0x000055878e5bcf65 in slot_tp_call (self=<_HookCaller(name='pytest_runtest_protocol', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd34875f0>, argnames=('item',), kwargnames=(), plugin=<module at remote 0x7fdbd3250ef0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='faulthandler', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd3254d80>, <HookImpl(function=<function at remote 0x7fdbd33b7ac0>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd33ee4b0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': False, 'specname': None}, plugin_name='unittest', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd33dfa80>, <HookImpl(function=<function at remote 0x7fdbd3643330>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd4139370>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': True, 'trylast': ...(truncated), args=(), kwds={'item': <Function at remote 0x7fdbd0a83f60>, 'nextitem': <Function(keywords=<NodeKeywords at remote 0x7fdbd0a87ac0>, own_markers=[], extra_keyword_matches=set(), stash=<Stash at remote 0x7fdbd0cc0190>, _report_sections=[], user_properties=[], originalname='test_set_listener', _obj=<method at remote 0x7fdbd0a9b770>, _fixtureinfo=<FuncFixtureInfo at remote 0x7fdbd0a9b6b0>, fixturenames=['mock_rest_bot', 'mock_interaction_server', 'mock_rest_client', 'mock_entity_factory', 'mock_http_settings', 'mock_proxy_settings', 'mock_executor'], funcargs={}, _request=<FixtureRequest(_pyfuncitem=<...>, fixturename=None, _scope=<Scope(_value_='function', _name_='Function', __objclass__=<EnumType(_generate_next_value_=<function at remote 0x7fdbd4109230>, __module__='_pytest.scope', __annotations__={'Function': '_ScopeName', 'Class': '_ScopeName', 'Module': '_ScopeName', 'Package': '_ScopeName', 'Session': '_ScopeName'}, __doc__='\n    Represents one of the possible fixture scopes in pytest.\n\n    Scopes are ordered from low...(truncated)) at Objects/typeobject.c:7630
#80 0x000055878e535764 in _PyObject_MakeTpCall (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<_HookCaller(name='pytest_runtest_protocol', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd34875f0>, argnames=('item',), kwargnames=(), plugin=<module at remote 0x7fdbd3250ef0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='faulthandler', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd3254d80>, <HookImpl(function=<function at remote 0x7fdbd33b7ac0>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd33ee4b0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': False, 'specname': None}, plugin_name='unittest', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd33dfa80>, <HookImpl(function=<function at remote 0x7fdbd3643330>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd4139370>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': True, 'trylast': ...(truncated), args=args@entry=0x7fdbd42d5890, nargs=<optimized out>, keywords=keywords@entry=('item', 'nextitem')) at Objects/call.c:214
#81 0x000055878e5361fb in _PyObject_VectorcallTstate (kwnames=('item', 'nextitem'), nargsf=9223372036854775808, args=0x7fdbd42d5890, callable=<_HookCaller(name='pytest_runtest_protocol', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd34875f0>, argnames=('item',), kwargnames=(), plugin=<module at remote 0x7fdbd3250ef0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='faulthandler', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd3254d80>, <HookImpl(function=<function at remote 0x7fdbd33b7ac0>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd33ee4b0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': False, 'specname': None}, plugin_name='unittest', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd33dfa80>, <HookImpl(function=<function at remote 0x7fdbd3643330>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd4139370>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': True, 'trylast': ...(truncated), tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:90
#82 PyObject_Vectorcall (callable=callable@entry=<_HookCaller(name='pytest_runtest_protocol', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd34875f0>, argnames=('item',), kwargnames=(), plugin=<module at remote 0x7fdbd3250ef0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='faulthandler', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd3254d80>, <HookImpl(function=<function at remote 0x7fdbd33b7ac0>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd33ee4b0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': False, 'specname': None}, plugin_name='unittest', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd33dfa80>, <HookImpl(function=<function at remote 0x7fdbd3643330>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd4139370>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': True, 'trylast': ...(truncated), args=args@entry=0x7fdbd42d5890, nargsf=<optimized out>, nargsf@entry=9223372036854775808, kwnames=kwnames@entry=('item', 'nextitem')) at Objects/call.c:299
#83 0x000055878e6576a2 in trace_call_function (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<_HookCaller(name='pytest_runtest_protocol', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd34875f0>, argnames=('item',), kwargnames=(), plugin=<module at remote 0x7fdbd3250ef0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='faulthandler', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd3254d80>, <HookImpl(function=<function at remote 0x7fdbd33b7ac0>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd33ee4b0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': False, 'specname': None}, plugin_name='unittest', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd33dfa80>, <HookImpl(function=<function at remote 0x7fdbd3643330>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd4139370>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': True, 'trylast': ...(truncated), args=args@entry=0x7fdbd42d5890, nargs=<optimized out>, kwnames=('item', 'nextitem')) at Python/ceval.c:7305
#84 0x000055878e667ca9 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=frame@entry=0x7fdbd42d5810, throwflag=throwflag@entry=0) at Python/ceval.c:4767
#85 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d5810, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#86 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#87 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393
#88 0x000055878e534f24 in _PyVectorcall_Call (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=0x55878e53541c <_PyFunction_Vectorcall>, callable=callable@entry=<function at remote 0x7fdbd3637120>, tuple=tuple@entry=(<Session at remote 0x7fdbd25c6610>,), kwargs=kwargs@entry=0x0) at ./Include/object.h:772
#89 0x000055878e53538a in _PyObject_Call (tstate=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3637120>, args=args@entry=(<Session at remote 0x7fdbd25c6610>,), kwargs=kwargs@entry=0x0) at Objects/call.c:328
#90 0x000055878e5353f2 in PyObject_Call (callable=callable@entry=<function at remote 0x7fdbd3637120>, args=args@entry=(<Session at remote 0x7fdbd25c6610>,), kwargs=kwargs@entry=0x0) at Objects/call.c:355
#91 0x000055878e65797c in do_call_core (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<function at remote 0x7fdbd3637120>, callargs=callargs@entry=(<Session at remote 0x7fdbd25c6610>,), kwdict=kwdict@entry=0x0, use_tracing=255) at Python/ceval.c:7347
#92 0x000055878e66b6f1 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d5720, frame@entry=0x7fdbd42d55d8, throwflag=throwflag@entry=0) at Python/ceval.c:5377
#93 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d55d8, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#94 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#95 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393
#96 0x000055878e535a72 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3e18100>, args=args@entry=0x7ffdc69ab3a0, nargsf=nargsf@entry=1, kwargs=kwargs@entry={'session': <Session at remote 0x7fdbd25c6610>}) at Objects/call.c:152
#97 0x000055878e535c31 in _PyObject_Call_Prepend (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3e18100>, obj=obj@entry=<_HookCaller(name='pytest_runtestloop', _wrappers=[<HookImpl(function=<method at remote 0x7fdbd2733bf0>, argnames=('session',), kwargnames=(), plugin=<CovPlugin(pid=32382, cov_controller=<Central(cov_source=None, cov_report={}, cov_config='.coveragerc', cov_append=False, cov_branch=None, config=<Config() at remote 0x7fdbd26d46a0>, nodeid=None, cov=<Coverage(_no_disk=False, config=<CoverageConfig(attempted_config_files=['.coveragerc', 'setup.cfg', 'tox.ini', 'pyproject.toml'], config_files_read=['/home/brandtbucher/hikari/pyproject.toml'], config_file='/home/brandtbucher/hikari/pyproject.toml', _config_contents=b'# Copyright (c) 2020 Nekokatt\n# Copyright (c) 2021-present davfsa\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the "Software"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the...(truncated), args=args@entry=(), kwargs=kwargs@entry={'session': <Session at remote 0x7fdbd25c6610>}) at Objects/call.c:482
#98 0x000055878e5bcf65 in slot_tp_call (self=<_HookCaller(name='pytest_runtestloop', _wrappers=[<HookImpl(function=<method at remote 0x7fdbd2733bf0>, argnames=('session',), kwargnames=(), plugin=<CovPlugin(pid=32382, cov_controller=<Central(cov_source=None, cov_report={}, cov_config='.coveragerc', cov_append=False, cov_branch=None, config=<Config() at remote 0x7fdbd26d46a0>, nodeid=None, cov=<Coverage(_no_disk=False, config=<CoverageConfig(attempted_config_files=['.coveragerc', 'setup.cfg', 'tox.ini', 'pyproject.toml'], config_files_read=['/home/brandtbucher/hikari/pyproject.toml'], config_file='/home/brandtbucher/hikari/pyproject.toml', _config_contents=b'# Copyright (c) 2020 Nekokatt\n# Copyright (c) 2021-present davfsa\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the "Software"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the...(truncated), args=(), kwds={'session': <Session at remote 0x7fdbd25c6610>}) at Objects/typeobject.c:7630
#99 0x000055878e535764 in _PyObject_MakeTpCall (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<_HookCaller(name='pytest_runtestloop', _wrappers=[<HookImpl(function=<method at remote 0x7fdbd2733bf0>, argnames=('session',), kwargnames=(), plugin=<CovPlugin(pid=32382, cov_controller=<Central(cov_source=None, cov_report={}, cov_config='.coveragerc', cov_append=False, cov_branch=None, config=<Config() at remote 0x7fdbd26d46a0>, nodeid=None, cov=<Coverage(_no_disk=False, config=<CoverageConfig(attempted_config_files=['.coveragerc', 'setup.cfg', 'tox.ini', 'pyproject.toml'], config_files_read=['/home/brandtbucher/hikari/pyproject.toml'], config_file='/home/brandtbucher/hikari/pyproject.toml', _config_contents=b'# Copyright (c) 2020 Nekokatt\n# Copyright (c) 2021-present davfsa\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the "Software"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the...(truncated), args=args@entry=0x7fdbd42d55d0, nargs=<optimized out>, keywords=keywords@entry=('session',)) at Objects/call.c:214
#100 0x000055878e5361fb in _PyObject_VectorcallTstate (kwnames=('session',), nargsf=9223372036854775808, args=0x7fdbd42d55d0, callable=<_HookCaller(name='pytest_runtestloop', _wrappers=[<HookImpl(function=<method at remote 0x7fdbd2733bf0>, argnames=('session',), kwargnames=(), plugin=<CovPlugin(pid=32382, cov_controller=<Central(cov_source=None, cov_report={}, cov_config='.coveragerc', cov_append=False, cov_branch=None, config=<Config() at remote 0x7fdbd26d46a0>, nodeid=None, cov=<Coverage(_no_disk=False, config=<CoverageConfig(attempted_config_files=['.coveragerc', 'setup.cfg', 'tox.ini', 'pyproject.toml'], config_files_read=['/home/brandtbucher/hikari/pyproject.toml'], config_file='/home/brandtbucher/hikari/pyproject.toml', _config_contents=b'# Copyright (c) 2020 Nekokatt\n# Copyright (c) 2021-present davfsa\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the "Software"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the...(truncated), tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:90
#101 PyObject_Vectorcall (callable=callable@entry=<_HookCaller(name='pytest_runtestloop', _wrappers=[<HookImpl(function=<method at remote 0x7fdbd2733bf0>, argnames=('session',), kwargnames=(), plugin=<CovPlugin(pid=32382, cov_controller=<Central(cov_source=None, cov_report={}, cov_config='.coveragerc', cov_append=False, cov_branch=None, config=<Config() at remote 0x7fdbd26d46a0>, nodeid=None, cov=<Coverage(_no_disk=False, config=<CoverageConfig(attempted_config_files=['.coveragerc', 'setup.cfg', 'tox.ini', 'pyproject.toml'], config_files_read=['/home/brandtbucher/hikari/pyproject.toml'], config_file='/home/brandtbucher/hikari/pyproject.toml', _config_contents=b'# Copyright (c) 2020 Nekokatt\n# Copyright (c) 2021-present davfsa\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the "Software"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the...(truncated), args=args@entry=0x7fdbd42d55d0, nargsf=<optimized out>, nargsf@entry=9223372036854775808, kwnames=kwnames@entry=('session',)) at Objects/call.c:299
#102 0x000055878e6576a2 in trace_call_function (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<_HookCaller(name='pytest_runtestloop', _wrappers=[<HookImpl(function=<method at remote 0x7fdbd2733bf0>, argnames=('session',), kwargnames=(), plugin=<CovPlugin(pid=32382, cov_controller=<Central(cov_source=None, cov_report={}, cov_config='.coveragerc', cov_append=False, cov_branch=None, config=<Config() at remote 0x7fdbd26d46a0>, nodeid=None, cov=<Coverage(_no_disk=False, config=<CoverageConfig(attempted_config_files=['.coveragerc', 'setup.cfg', 'tox.ini', 'pyproject.toml'], config_files_read=['/home/brandtbucher/hikari/pyproject.toml'], config_file='/home/brandtbucher/hikari/pyproject.toml', _config_contents=b'# Copyright (c) 2020 Nekokatt\n# Copyright (c) 2021-present davfsa\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the "Software"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the...(truncated), args=args@entry=0x7fdbd42d55d0, nargs=<optimized out>, kwnames=('session',)) at Python/ceval.c:7305
#103 0x000055878e667ca9 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d5568, frame@entry=0x7fdbd42d5430, throwflag=throwflag@entry=0) at Python/ceval.c:4767
#104 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d5430, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#105 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#106 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393
#107 0x000055878e534f24 in _PyVectorcall_Call (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=0x55878e53541c <_PyFunction_Vectorcall>, callable=callable@entry=<function at remote 0x7fdbd3636f10>, tuple=tuple@entry=(<Config(option=<Namespace(keyword='', markexpr='', maxfail=0, continue_on_collection_errors=False, confcutdir=None, noconftest=False, keepduplicates=False, collect_in_virtualenv=False, importmode='prepend', basetemp=None, durations=None, durations_min=<float at remote 0x7fdbd43ce4a0>, version=0, plugins=[], traceconfig=False, showfixtures=False, show_fixtures_per_test=False, verbose=1, no_header=False, no_summary=False, reportchars='fE', disable_warnings=False, showlocals=False, tbstyle='auto', showcapture='all', fulltrace=False, color='auto', code_highlight='yes', capture='fd', runxfail=False, pastebin=None, assertmode='rewrite', xmlpath=None, junitprefix=None, doctestmodules=False, doctestreport='udiff', doctestglob=[], doctest_ignore_import_errors=False, doctest_continue_on_failure=False, last_failed_no_failures='all', stepwise=False, stepwise_skip=False, cov_source=[True], cov_report={'': None}, cov_config='.coveragerc', no_cov_on_fail=False, no_cov=False, cov_append=False, cov_branch=None, asyncio_mode=...(truncated), kwargs=kwargs@entry=0x0) at ./Include/object.h:772
#108 0x000055878e53538a in _PyObject_Call (tstate=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3636f10>, args=args@entry=(<Config(option=<Namespace(keyword='', markexpr='', maxfail=0, continue_on_collection_errors=False, confcutdir=None, noconftest=False, keepduplicates=False, collect_in_virtualenv=False, importmode='prepend', basetemp=None, durations=None, durations_min=<float at remote 0x7fdbd43ce4a0>, version=0, plugins=[], traceconfig=False, showfixtures=False, show_fixtures_per_test=False, verbose=1, no_header=False, no_summary=False, reportchars='fE', disable_warnings=False, showlocals=False, tbstyle='auto', showcapture='all', fulltrace=False, color='auto', code_highlight='yes', capture='fd', runxfail=False, pastebin=None, assertmode='rewrite', xmlpath=None, junitprefix=None, doctestmodules=False, doctestreport='udiff', doctestglob=[], doctest_ignore_import_errors=False, doctest_continue_on_failure=False, last_failed_no_failures='all', stepwise=False, stepwise_skip=False, cov_source=[True], cov_report={'': None}, cov_config='.coveragerc', no_cov_on_fail=False, no_cov=False, cov_append=False, cov_branch=None, asyncio_mode=...(truncated), kwargs=kwargs@entry=0x0) at Objects/call.c:328
#109 0x000055878e5353f2 in PyObject_Call (callable=callable@entry=<function at remote 0x7fdbd3636f10>, args=args@entry=(<Config(option=<Namespace(keyword='', markexpr='', maxfail=0, continue_on_collection_errors=False, confcutdir=None, noconftest=False, keepduplicates=False, collect_in_virtualenv=False, importmode='prepend', basetemp=None, durations=None, durations_min=<float at remote 0x7fdbd43ce4a0>, version=0, plugins=[], traceconfig=False, showfixtures=False, show_fixtures_per_test=False, verbose=1, no_header=False, no_summary=False, reportchars='fE', disable_warnings=False, showlocals=False, tbstyle='auto', showcapture='all', fulltrace=False, color='auto', code_highlight='yes', capture='fd', runxfail=False, pastebin=None, assertmode='rewrite', xmlpath=None, junitprefix=None, doctestmodules=False, doctestreport='udiff', doctestglob=[], doctest_ignore_import_errors=False, doctest_continue_on_failure=False, last_failed_no_failures='all', stepwise=False, stepwise_skip=False, cov_source=[True], cov_report={'': None}, cov_config='.coveragerc', no_cov_on_fail=False, no_cov=False, cov_append=False, cov_branch=None, asyncio_mode=...(truncated), kwargs=kwargs@entry=0x0) at Objects/call.c:355
#110 0x000055878e65797c in do_call_core (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<function at remote 0x7fdbd3636f10>, callargs=callargs@entry=(<Config(option=<Namespace(keyword='', markexpr='', maxfail=0, continue_on_collection_errors=False, confcutdir=None, noconftest=False, keepduplicates=False, collect_in_virtualenv=False, importmode='prepend', basetemp=None, durations=None, durations_min=<float at remote 0x7fdbd43ce4a0>, version=0, plugins=[], traceconfig=False, showfixtures=False, show_fixtures_per_test=False, verbose=1, no_header=False, no_summary=False, reportchars='fE', disable_warnings=False, showlocals=False, tbstyle='auto', showcapture='all', fulltrace=False, color='auto', code_highlight='yes', capture='fd', runxfail=False, pastebin=None, assertmode='rewrite', xmlpath=None, junitprefix=None, doctestmodules=False, doctestreport='udiff', doctestglob=[], doctest_ignore_import_errors=False, doctest_continue_on_failure=False, last_failed_no_failures='all', stepwise=False, stepwise_skip=False, cov_source=[True], cov_report={'': None}, cov_config='.coveragerc', no_cov_on_fail=False, no_cov=False, cov_append=False, cov_branch=None, asyncio_mode=...(truncated), kwdict=kwdict@entry=0x0, use_tracing=255) at Python/ceval.c:7347
#111 0x000055878e66b6f1 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d5340, frame@entry=0x7fdbd42d51f8, throwflag=throwflag@entry=0) at Python/ceval.c:5377
#112 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d51f8, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#113 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#114 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393
#115 0x000055878e535a72 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3e18100>, args=args@entry=0x7ffdc69ab970, nargsf=nargsf@entry=1, kwargs=kwargs@entry={'config': <Config(option=<Namespace(keyword='', markexpr='', maxfail=0, continue_on_collection_errors=False, confcutdir=None, noconftest=False, keepduplicates=False, collect_in_virtualenv=False, importmode='prepend', basetemp=None, durations=None, durations_min=<float at remote 0x7fdbd43ce4a0>, version=0, plugins=[], traceconfig=False, showfixtures=False, show_fixtures_per_test=False, verbose=1, no_header=False, no_summary=False, reportchars='fE', disable_warnings=False, showlocals=False, tbstyle='auto', showcapture='all', fulltrace=False, color='auto', code_highlight='yes', capture='fd', runxfail=False, pastebin=None, assertmode='rewrite', xmlpath=None, junitprefix=None, doctestmodules=False, doctestreport='udiff', doctestglob=[], doctest_ignore_import_errors=False, doctest_continue_on_failure=False, last_failed_no_failures='all', stepwise=False, stepwise_skip=False, cov_source=[True], cov_report={'': None}, cov_config='.coveragerc', no_cov_on_fail=False, no_cov=False, cov_append=False, cov_branch=None, asy...(truncated)) at Objects/call.c:152
#116 0x000055878e535c31 in _PyObject_Call_Prepend (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3e18100>, obj=obj@entry=<_HookCaller(name='pytest_cmdline_main', _wrappers=[], _nonwrappers=[<HookImpl(function=<function at remote 0x7fdbd3636f10>, argnames=('config',), kwargnames=(), plugin=<module at remote 0x7fdbd3898530>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='main', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33ce950>, <HookImpl(function=<function at remote 0x7fdbd33b6780>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd33da930>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='helpconfig', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33dcc90>, <HookImpl(function=<function at remote 0x7fdbd3655390>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd363ff50>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': F...(truncated), args=args@entry=(), kwargs=kwargs@entry={'config': <Config(option=<Namespace(keyword='', markexpr='', maxfail=0, continue_on_collection_errors=False, confcutdir=None, noconftest=False, keepduplicates=False, collect_in_virtualenv=False, importmode='prepend', basetemp=None, durations=None, durations_min=<float at remote 0x7fdbd43ce4a0>, version=0, plugins=[], traceconfig=False, showfixtures=False, show_fixtures_per_test=False, verbose=1, no_header=False, no_summary=False, reportchars='fE', disable_warnings=False, showlocals=False, tbstyle='auto', showcapture='all', fulltrace=False, color='auto', code_highlight='yes', capture='fd', runxfail=False, pastebin=None, assertmode='rewrite', xmlpath=None, junitprefix=None, doctestmodules=False, doctestreport='udiff', doctestglob=[], doctest_ignore_import_errors=False, doctest_continue_on_failure=False, last_failed_no_failures='all', stepwise=False, stepwise_skip=False, cov_source=[True], cov_report={'': None}, cov_config='.coveragerc', no_cov_on_fail=False, no_cov=False, cov_append=False, cov_branch=None, asy...(truncated)) at Objects/call.c:482
#117 0x000055878e5bcf65 in slot_tp_call (self=<_HookCaller(name='pytest_cmdline_main', _wrappers=[], _nonwrappers=[<HookImpl(function=<function at remote 0x7fdbd3636f10>, argnames=('config',), kwargnames=(), plugin=<module at remote 0x7fdbd3898530>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='main', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33ce950>, <HookImpl(function=<function at remote 0x7fdbd33b6780>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd33da930>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='helpconfig', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33dcc90>, <HookImpl(function=<function at remote 0x7fdbd3655390>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd363ff50>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': F...(truncated), args=(), kwds={'config': <Config(option=<Namespace(keyword='', markexpr='', maxfail=0, continue_on_collection_errors=False, confcutdir=None, noconftest=False, keepduplicates=False, collect_in_virtualenv=False, importmode='prepend', basetemp=None, durations=None, durations_min=<float at remote 0x7fdbd43ce4a0>, version=0, plugins=[], traceconfig=False, showfixtures=False, show_fixtures_per_test=False, verbose=1, no_header=False, no_summary=False, reportchars='fE', disable_warnings=False, showlocals=False, tbstyle='auto', showcapture='all', fulltrace=False, color='auto', code_highlight='yes', capture='fd', runxfail=False, pastebin=None, assertmode='rewrite', xmlpath=None, junitprefix=None, doctestmodules=False, doctestreport='udiff', doctestglob=[], doctest_ignore_import_errors=False, doctest_continue_on_failure=False, last_failed_no_failures='all', stepwise=False, stepwise_skip=False, cov_source=[True], cov_report={'': None}, cov_config='.coveragerc', no_cov_on_fail=False, no_cov=False, cov_append=False, cov_branch=None, asy...(truncated)) at Objects/typeobject.c:7630
#118 0x000055878e535764 in _PyObject_MakeTpCall (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<_HookCaller(name='pytest_cmdline_main', _wrappers=[], _nonwrappers=[<HookImpl(function=<function at remote 0x7fdbd3636f10>, argnames=('config',), kwargnames=(), plugin=<module at remote 0x7fdbd3898530>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='main', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33ce950>, <HookImpl(function=<function at remote 0x7fdbd33b6780>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd33da930>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='helpconfig', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33dcc90>, <HookImpl(function=<function at remote 0x7fdbd3655390>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd363ff50>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': F...(truncated), args=args@entry=0x7fdbd42d51d0, nargs=<optimized out>, keywords=keywords@entry=('config',)) at Objects/call.c:214
#119 0x000055878e5361fb in _PyObject_VectorcallTstate (kwnames=('config',), nargsf=9223372036854775808, args=0x7fdbd42d51d0, callable=<_HookCaller(name='pytest_cmdline_main', _wrappers=[], _nonwrappers=[<HookImpl(function=<function at remote 0x7fdbd3636f10>, argnames=('config',), kwargnames=(), plugin=<module at remote 0x7fdbd3898530>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='main', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33ce950>, <HookImpl(function=<function at remote 0x7fdbd33b6780>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd33da930>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='helpconfig', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33dcc90>, <HookImpl(function=<function at remote 0x7fdbd3655390>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd363ff50>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': F...(truncated), tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:90
#120 PyObject_Vectorcall (callable=callable@entry=<_HookCaller(name='pytest_cmdline_main', _wrappers=[], _nonwrappers=[<HookImpl(function=<function at remote 0x7fdbd3636f10>, argnames=('config',), kwargnames=(), plugin=<module at remote 0x7fdbd3898530>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='main', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33ce950>, <HookImpl(function=<function at remote 0x7fdbd33b6780>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd33da930>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='helpconfig', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33dcc90>, <HookImpl(function=<function at remote 0x7fdbd3655390>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd363ff50>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': F...(truncated), args=args@entry=0x7fdbd42d51d0, nargsf=<optimized out>, nargsf@entry=9223372036854775808, kwnames=kwnames@entry=('config',)) at Objects/call.c:299
#121 0x000055878e6576a2 in trace_call_function (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<_HookCaller(name='pytest_cmdline_main', _wrappers=[], _nonwrappers=[<HookImpl(function=<function at remote 0x7fdbd3636f10>, argnames=('config',), kwargnames=(), plugin=<module at remote 0x7fdbd3898530>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='main', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33ce950>, <HookImpl(function=<function at remote 0x7fdbd33b6780>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd33da930>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='helpconfig', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33dcc90>, <HookImpl(function=<function at remote 0x7fdbd3655390>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd363ff50>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': F...(truncated), args=args@entry=0x7fdbd42d51d0, nargs=<optimized out>, kwnames=('config',)) at Python/ceval.c:7305
#122 0x000055878e667ca9 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d5120, frame@entry=0x7fdbd42d5020, throwflag=throwflag@entry=0) at Python/ceval.c:4767
#123 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d5020, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#124 _PyEval_Vector (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=0x7fdbd41fa620, locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:6428
#125 0x000055878e66d81b in PyEval_EvalCode (co=co@entry=<code at remote 0x7fdbd437cf40>, globals=globals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}, locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}) at Python/ceval.c:1154
#126 0x000055878e6c088b in run_eval_code_obj (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, co=co@entry=0x7fdbd437cf40, globals=globals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}, locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}) at Python/pythonrun.c:1714
#127 0x000055878e6c1028 in run_mod (mod=mod@entry=0x55878f87d5b0, filename=filename@entry='/home/brandtbucher/env311/bin/pytest', globals=globals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}, locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}, flags=flags@entry=0x7ffdc69abe28, arena=arena@entry=0x7fdbd42731c0) at Python/pythonrun.c:1735
#128 0x000055878e6c1134 in pyrun_file (fp=fp@entry=0x55878f870090, filename=filename@entry='/home/brandtbucher/env311/bin/pytest', start=start@entry=257, globals=globals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}, locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}, closeit=closeit@entry=1, flags=0x7ffdc69abe28) at Python/pythonrun.c:1630
#129 0x000055878e6c4642 in _PyRun_SimpleFileObject (fp=fp@entry=0x55878f870090, filename=filename@entry='/home/brandtbucher/env311/bin/pytest', closeit=closeit@entry=1, flags=flags@entry=0x7ffdc69abe28) at Python/pythonrun.c:440
#130 0x000055878e6c488e in _PyRun_AnyFileObject (fp=fp@entry=0x55878f870090, filename=filename@entry='/home/brandtbucher/env311/bin/pytest', closeit=closeit@entry=1, flags=flags@entry=0x7ffdc69abe28) at Python/pythonrun.c:79
#131 0x000055878e6e808b in pymain_run_file_obj (program_name=program_name@entry='/home/brandtbucher/env311/bin/python', filename=filename@entry='/home/brandtbucher/env311/bin/pytest', skip_source_first_line=0) at Modules/main.c:360
#132 0x000055878e6e83e3 in pymain_run_file (config=config@entry=0x55878e9f2fc0 <_PyRuntime+59904>) at Modules/main.c:379
#133 0x000055878e6e94b8 in pymain_run_python (exitcode=exitcode@entry=0x7ffdc69abf84) at Modules/main.c:601
#134 0x000055878e6e9522 in Py_RunMain () at Modules/main.c:680
#135 0x000055878e6e959c in pymain_main (args=args@entry=0x7ffdc69abfe0) at Modules/main.c:710
#136 0x000055878e6e9668 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:734
#137 0x000055878e496d06 in main (argc=<optimized out>, argv=<optimized out>) at ./Programs/python.c:15

@brandtbucher
Copy link
Member

brandtbucher commented Sep 22, 2022 •
edited
Loading

Confirmed that it's not coverage. Setting a simple trace function at the top of the file is enough to make things crash:

def trace(frame, event, arg):
    return trace

sys.settrace(trace)

@brandtbucher
Copy link
Member

Not sure if it's a symptom or the bug, but __lltrace__ tells me that the RETURN_VALUE (83) opcode in my trace function is being incremented at some point (just before the crash) and executed as IMPORT_STAR (84). The VM then runs off the end of the bytecode, prints that it's reached an unknown opcode, and then I get a segfault during a garbage collection sometime later.

I'm fairly confident that the specialization machinery isn't responsible for the changing opcode (I disabled specialization of LOAD_GLOBAL, and even tried another version of the trace function that used LOAD_FAST... same behavior). Pretty bizarre, so far, but I feel like I'm at least making some progress.

@brandtbucher
Copy link
Member

Another puzzle piece: the segfault only happens if the file is loaded from a cached PYC. Forcing recompilation with PYTHONDONTWRITEBYTECODE=1 seems to get rid of the crash, and if you modify the file manually, the first run (not cached) will succeed, but the rest (cached) will crash.

@brandtbucher
Copy link
Member

Again, these could just be action at a distance from the VM getting into some bad state earlier in the program, or changing when and how GC/tracing events happen. But they're interesting, nonetheless.

@davfsa
Copy link
Author

davfsa commented Sep 23, 2022 •
edited
Loading

Sorry for not being able to send this yesterday, was exhausted. But this morning i started playing around a bit with this to try and get a core dump. Instead of a core dump, I got this rather interesting error:

Full traceback 
exc = <class 'ResourceWarning'>, value = ResourceWarning('unclosed <socket.socket fd=13, family=1, type=1, proto=0>'), tb = <traceback object at 0x7fd30f0bbe00>, limit = None, chain = True

    def format_exception(exc, /, value=_sentinel, tb=_sentinel, limit=None, \
                         chain=True):
        """Format a stack trace and the exception information.
    
        The arguments have the same meaning as the corresponding arguments
        to print_exception().  The return value is a list of strings, each
        ending in a newline and some containing internal newlines.  When
        these lines are concatenated and printed, exactly the same text is
        printed as does print_exception().
        """
        value, tb = _parse_value_tb(exc, value, tb)
>       te = TracebackException(type(value), value, tb, limit=limit, compact=True)

chain      = True
exc        = <class 'ResourceWarning'>
limit      = None
tb         = <traceback object at 0x7fd30f0bbe00>
value      = ResourceWarning('unclosed <socket.socket fd=13, family=1, type=1, proto=0>')

/usr/lib/python3.11/traceback.py:139: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python3.11/traceback.py:688: in __init__
    self.stack = StackSummary._extract_from_extended_frame_gen(
        _seen      = {140544466309120}
        capture_locals = False
        compact    = True
        exc_traceback = <traceback object at 0x7fd30f0bbe00>
        exc_type   = <class 'ResourceWarning'>
        exc_value  = ResourceWarning('unclosed <socket.socket fd=13, family=1, type=1, proto=0>')
        is_recursive_call = False
        limit      = None
        lookup_lines = True
        max_group_depth = 10
        max_group_width = 15
        self       = <traceback.TracebackException object at 0x7fd30fc00450>
/usr/lib/python3.11/traceback.py:416: in _extract_from_extended_frame_gen
    for f, (lineno, end_lineno, colno, end_colno) in frame_gen:
        capture_locals = False
        fnames     = set()
        frame_gen  = <generator object _walk_tb_with_full_positions at 0x7fd30f2f89e0>
        klass      = <class 'traceback.StackSummary'>
        limit      = None
        lookup_lines = True
        result     = []
/usr/lib/python3.11/traceback.py:353: in _walk_tb_with_full_positions
    positions = _get_code_position(tb.tb_frame.f_code, tb.tb_lasti)
        tb         = <traceback object at 0x7fd30f0bbe00>
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

code = <Mock name='mock.close()' id='140544468863376'>, instruction_index = 0

    def _get_code_position(code, instruction_index):
        if instruction_index < 0:
            return (None, None, None, None)
        positions_gen = code.co_positions()
>       return next(itertools.islice(positions_gen, instruction_index // 2, None))
E       TypeError: 'Mock' object is not iterable

code       = <Mock name='mock.close()' id='140544468863376'>
instruction_index = 0
positions_gen = <Mock name='mock.close().co_positions()' id='140544468863440'>

/usr/lib/python3.11/traceback.py:367: TypeError

The first part of the traceback is normal. It is possible aiohttp (a library we use) to leave unclosed sockets if not closed correctly. Which we dont mind for the sake of testing. But the interesting part is the last part of the traceback, where somehow code becomes a unnitest.mock.Mock. This caused .co_positions to not fail, as it will just return a mock object, but the later usage with itertools did. So, somehow, tb.tb_frame.f_code is a mock (assuming that it is gotten from one of the objects in our test suite), but that doesnt look quite right to me. I'll keep playing around with it and posting my findings.

Thank you @brandtbucher for having a look at it yesterday!

EDIT:

Dont want to spam your inbox, so just appending this here, but I have managed to find another case of something seemingly random being injected as code.

Full traceback 
exc = <class 'ResourceWarning'>, value = ResourceWarning('unclosed <socket.socket fd=13, family=1, type=1, proto=0>'), tb = <traceback object at 0x7f0d233e1180>, limit = None, chain = True

    def format_exception(exc, /, value=_sentinel, tb=_sentinel, limit=None, \
                         chain=True):
        """Format a stack trace and the exception information.
    
        The arguments have the same meaning as the corresponding arguments
        to print_exception().  The return value is a list of strings, each
        ending in a newline and some containing internal newlines.  When
        these lines are concatenated and printed, exactly the same text is
        printed as does print_exception().
        """
        value, tb = _parse_value_tb(exc, value, tb)
>       te = TracebackException(type(value), value, tb, limit=limit, compact=True)

chain      = True
exc        = <class 'ResourceWarning'>
limit      = None
tb         = <traceback object at 0x7f0d233e1180>
value      = ResourceWarning('unclosed <socket.socket fd=13, family=1, type=1, proto=0>')

/usr/lib/python3.11/traceback.py:139: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python3.11/traceback.py:688: in __init__
    self.stack = StackSummary._extract_from_extended_frame_gen(
        _seen      = {139694401470720}
        capture_locals = False
        compact    = True
        exc_traceback = <traceback object at 0x7f0d233e1180>
        exc_type   = <class 'ResourceWarning'>
        exc_value  = ResourceWarning('unclosed <socket.socket fd=13, family=1, type=1, proto=0>')
        is_recursive_call = False
        limit      = None
        lookup_lines = True
        max_group_depth = 10
        max_group_width = 15
        self       = <traceback.TracebackException object at 0x7f0d233fbe50>
/usr/lib/python3.11/traceback.py:416: in _extract_from_extended_frame_gen
    for f, (lineno, end_lineno, colno, end_colno) in frame_gen:
        capture_locals = False
        fnames     = set()
        frame_gen  = <generator object _walk_tb_with_full_positions at 0x7f0d2364f220>
        klass      = <class 'traceback.StackSummary'>
        limit      = None
        lookup_lines = True
        result     = []
/usr/lib/python3.11/traceback.py:353: in _walk_tb_with_full_positions
    positions = _get_code_position(tb.tb_frame.f_code, tb.tb_lasti)
        tb         = <traceback object at 0x7f0d233e1180>
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

code = {'@py_builtins': <module 'builtins' (built-in)>, '@pytest_ar': <module '_pytest.assertion.rewrite' from '/home/davfsa/...ass 'AssertionError'>, 'AttributeError': <class 'AttributeError'>, 'BaseException': <class 'BaseException'>, ...}, ...}
instruction_index = 0

    def _get_code_position(code, instruction_index):
        if instruction_index < 0:
            return (None, None, None, None)
>       positions_gen = code.co_positions()
E       AttributeError: 'dict' object has no attribute 'co_positions'

code       = {'@py_builtins': <module 'builtins' (built-in)>, '@pytest_ar': <module '_pytest.assertion.rewrite' from '/home/davfsa/...ass 'AssertionError'>, 'AttributeError': <class 'AttributeError'>, 'BaseException': <class 'BaseException'>, ...}, ...}
instruction_index = 0

/usr/lib/python3.11/traceback.py:366: AttributeError

@davfsa
Copy link
Author

davfsa commented Sep 23, 2022 •
edited
Loading

Another puzzle piece: the segfault only happens if the file is loaded from a cached PYC. Forcing recompilation with PYTHONDONTWRITEBYTECODE=1 seems to get rid of the crash, and if you modify the file manually, the first run (not cached) will succeed, but the rest (cached) will crash.

Can confirm this (not 100% of the time, but ill say 9/10 times), when running with already available pyc files, a segfault occurs (see bellow) in the same place all the time, but when I remove it and run the test suite again, a segfault happens in a different part of the run. Not modifying anything and running it again, the segfault doesnt happen at all. But two or three more runs later and the initial segfault can be found again the same place.

This seems to be really random and inconsistent for the most part. The tracebacks show that they are usually caused during the cleanup of an unclosed socket, which is controlled by GC and I dont think it can run in a predictable way outside of the current setup I have. I was having a go at trying to get a minimally reproducible project to make it easier to debug, but no luck.

If you are curious, I am getting the segfaults weirdness that I mentioned above by running the following:

python -m nox -s pytest -- --randomly-seed=2527224107

(We use a random-seed plugin to randomize the order of the tests in every run. So I tried to make it as consistent as possible. Also, python is python3.11-rc2)

Core dump 
#0  PyObject_GetAttr (v=v@entry=0x7f1084d07ab0, name=name@entry=0x7f10840bcb80) at ../Objects/object.c:916
#1  0x000055d50a427c0d in _PyObject_GetMethod (obj=obj@entry=0x7f1084d07ab0, name=0x7f10840bcb80, method=method@entry=0x7ffcd8fcc100) at ../Objects/object.c:1163
#2  0x000055d50a4f30b6 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=0x7f1084d075c8, frame@entry=0x7f107e9a7aa0, throwflag=throwflag@entry=0) at ../Python/ceval.c:4490
#3  0x000055d50a3e2278 in _PyEval_EvalFrame (throwflag=0, frame=0x7f107e9a7aa0, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#4  gen_send_ex2 (gen=0x7f107e9a7a50, arg=arg@entry=0x0, presult=presult@entry=0x7ffcd8fcc200, exc=exc@entry=0, closing=closing@entry=0) at ../Objects/genobject.c:219
#5  0x000055d50a3e2d2d in gen_iternext (gen=<optimized out>) at ../Objects/genobject.c:594
#6  0x000055d50a4f23c1 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=0x7f1084d07498, frame@entry=0x7f1084d07338, throwflag=throwflag@entry=0) at ../Python/ceval.c:4339
#7  0x000055d50a4fa0b2 in _PyEval_EvalFrame (throwflag=0, frame=0x7f1084d07338, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#8  _PyEval_Vector (tstate=0x55d50a895a38 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:6429
#9  0x000055d50a3c2ac0 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:393
#10 0x000055d50a3c3403 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083dccc00, args=args@entry=0x7ffcd8fcc420, nargsf=nargsf@entry=4, 
    kwargs=kwargs@entry=0x7f107f4d8530) at ../Objects/call.c:152
#11 0x000055d50a3c35d3 in _PyObject_Call_Prepend (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083dccc00, obj=obj@entry=0x7f107df320e0, args=args@entry=0x7f107e1de3f0, 
    kwargs=kwargs@entry=0x7f107f4d8530) at ../Objects/call.c:482
#12 0x000055d50a44ab4f in slot_tp_init (self=0x7f107df320e0, args=0x7f107e1de3f0, kwds=0x7f107f4d8530) at ../Objects/typeobject.c:7861
#13 0x000055d50a447ae9 in type_call (type=<optimized out>, args=0x7f107e1de3f0, kwds=0x7f107f4d8530) at ../Objects/typeobject.c:1112
#14 0x000055d50a3c2c93 in _PyObject_MakeTpCall (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x55d50b381af0, args=args@entry=0x7f1084d07310, nargs=<optimized out>, 
    keywords=keywords@entry=0x7f1083d84820) at ../Objects/call.c:214
#15 0x000055d50a3c304f in _PyObject_VectorcallTstate (tstate=0x55d50a895a38 <_PyRuntime+166328>, callable=0x55d50b381af0, args=0x7f1084d07310, nargsf=<optimized out>, kwnames=0x7f1083d84820) at ../Include/internal/pycore_call.h:90
#16 0x000055d50a3c3077 in PyObject_Vectorcall (callable=callable@entry=0x55d50b381af0, args=args@entry=0x7f1084d07310, nargsf=nargsf@entry=9223372036854775811, kwnames=kwnames@entry=0x7f1083d84820) at ../Objects/call.c:299
#17 0x000055d50a4e37b6 in trace_call_function (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=func@entry=0x55d50b381af0, args=args@entry=0x7f1084d07310, nargs=<optimized out>, kwnames=0x7f1083d84820)
    at ../Python/ceval.c:7309
#18 0x000055d50a4f42fc in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=0x7f1084d07288, frame@entry=0x7f107f1dfd60, throwflag=throwflag@entry=0) at ../Python/ceval.c:4766
#19 0x000055d50a3e2278 in _PyEval_EvalFrame (throwflag=0, frame=0x7f107f1dfd60, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#20 gen_send_ex2 (gen=gen@entry=0x7f107f1dfd10, arg=0x7f107f4bff80, presult=presult@entry=0x7ffcd8fcc780, exc=exc@entry=0, closing=closing@entry=0) at ../Objects/genobject.c:219
#21 0x000055d50a3e2659 in gen_send_ex (gen=0x7f107f1dfd10, arg=<optimized out>, exc=exc@entry=0, closing=closing@entry=0) at ../Objects/genobject.c:287
#22 0x000055d50a3e2cc6 in gen_send (gen=<optimized out>, arg=<optimized out>) at ../Objects/genobject.c:310
#23 0x000055d50a3d31fc in method_vectorcall_O (func=0x7f10840d7110, args=0x7ffcd8fcc950, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/descrobject.c:481
#24 0x000055d50a3c2f5e in _PyObject_VectorcallTstate (tstate=0x55d50a895a38 <_PyRuntime+166328>, callable=0x7f10840d7110, args=0x7ffcd8fcc950, nargsf=2, kwnames=0x0) at ../Include/internal/pycore_call.h:92
#25 0x000055d50a3c310f in PyObject_VectorcallMethod (name=<optimized out>, args=args@entry=0x7ffcd8fcc950, nargsf=<optimized out>, nargsf@entry=9223372036854775810, kwnames=kwnames@entry=0x0) at ../Objects/call.c:855
#26 0x000055d50a4e8cc7 in PyObject_CallMethodOneArg (arg=0x7f107f4bff80, name=<optimized out>, self=0x7f107f1dfd10) at ../Include/cpython/abstract.h:116
#27 _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=frame@entry=0x7f107e56f8b0, throwflag=throwflag@entry=0) at ../Python/ceval.c:2598
#28 0x000055d50a3e2278 in _PyEval_EvalFrame (throwflag=0, frame=0x7f107e56f8b0, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#29 gen_send_ex2 (gen=gen@entry=0x7f107e56f860, arg=0x7f107f4bff80, presult=presult@entry=0x7ffcd8fcca10, exc=exc@entry=0, closing=closing@entry=0) at ../Objects/genobject.c:219
#30 0x000055d50a3e2659 in gen_send_ex (gen=0x7f107e56f860, arg=<optimized out>, exc=exc@entry=0, closing=closing@entry=0) at ../Objects/genobject.c:287
#31 0x000055d50a3e2cc6 in gen_send (gen=<optimized out>, arg=<optimized out>) at ../Objects/genobject.c:310
#32 0x000055d50a420109 in cfunction_vectorcall_O (func=0x7f107dfbb950, args=0x7f1084d07268, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/methodobject.c:514
#33 0x000055d50a3c2f5e in _PyObject_VectorcallTstate (tstate=0x55d50a895a38 <_PyRuntime+166328>, callable=0x7f107dfbb950, args=0x7f1084d07268, nargsf=1, kwnames=0x0) at ../Include/internal/pycore_call.h:92
#34 0x000055d50a3c3077 in PyObject_Vectorcall (callable=callable@entry=0x7f107dfbb950, args=args@entry=0x7f1084d07268, nargsf=nargsf@entry=1, kwnames=kwnames@entry=0x0) at ../Objects/call.c:299
#35 0x000055d50a4e3a1c in trace_call_function (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=0x7f107dfbb950, func@entry=0x7f10840d7110, args=args@entry=0x7f1084d07260, nargs=2, kwnames=0x0) at ../Python/ceval.c:7303
#36 0x000055d50a4f42fc in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=0x7f1084d07198, frame@entry=0x7f1084d07050, throwflag=throwflag@entry=0) at ../Python/ceval.c:4766
#37 0x000055d50a4fa0b2 in _PyEval_EvalFrame (throwflag=0, frame=0x7f1084d07050, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#38 _PyEval_Vector (tstate=0x55d50a895a38 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:6429
#39 0x000055d50a3c2ac0 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:393
#40 0x000055d50a3c3403 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083b49230, args=args@entry=0x7ffcd8fccd10, nargsf=nargsf@entry=1, 
    kwargs=kwargs@entry=0x7f107f4d8b30) at ../Objects/call.c:152
#41 0x000055d50a3c35d3 in _PyObject_Call_Prepend (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083b49230, obj=obj@entry=0x7f1083181960, args=args@entry=0x55d50a87b698 <_PyRuntime+58904>, 
    kwargs=kwargs@entry=0x7f107f4d8b30) at ../Objects/call.c:482
#42 0x000055d50a44ad13 in slot_tp_call (self=0x7f1083181960, args=0x55d50a87b698 <_PyRuntime+58904>, kwds=0x7f107f4d8b30) at ../Objects/typeobject.c:7630
#43 0x000055d50a3c2914 in _PyObject_Call (tstate=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083181960, args=args@entry=0x55d50a87b698 <_PyRuntime+58904>, kwargs=kwargs@entry=0x7f107f4d8b30)
    at ../Objects/call.c:343
#44 0x000055d50a3c2a42 in PyObject_Call (callable=callable@entry=0x7f1083181960, args=args@entry=0x55d50a87b698 <_PyRuntime+58904>, kwargs=kwargs@entry=0x7f107f4d8b30) at ../Objects/call.c:355
#45 0x000055d50a4e3a90 in do_call_core (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=func@entry=0x7f1083181960, callargs=callargs@entry=0x55d50a87b698 <_PyRuntime+58904>, kwdict=kwdict@entry=0x7f107f4d8b30, 
    use_tracing=255) at ../Python/ceval.c:7351
#46 0x000055d50a4f7fe9 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=0x7f1084d06fc8, frame@entry=0x7f1084d06e48, throwflag=throwflag@entry=0) at ../Python/ceval.c:5376
#47 0x000055d50a4fa0b2 in _PyEval_EvalFrame (throwflag=0, frame=0x7f1084d06e48, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#48 _PyEval_Vector (tstate=0x55d50a895a38 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:6429
#49 0x000055d50a3c2ac0 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:393
#50 0x000055d50a3c2521 in _PyVectorcall_Call (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=0x55d50a3c2a6c <_PyFunction_Vectorcall>, callable=callable@entry=0x7f108338e360, tuple=tuple@entry=0x7f107e054190, 
    kwargs=kwargs@entry=0x7f107f4ba450) at ../Objects/call.c:245
#51 0x000055d50a3c29d5 in _PyObject_Call (tstate=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f108338e360, args=args@entry=0x7f107e054190, kwargs=kwargs@entry=0x7f107f4ba450) at ../Objects/call.c:328
#52 0x000055d50a3c2a42 in PyObject_Call (callable=callable@entry=0x7f108338e360, args=args@entry=0x7f107e054190, kwargs=kwargs@entry=0x7f107f4ba450) at ../Objects/call.c:355
#53 0x000055d50a4e3a90 in do_call_core (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=func@entry=0x7f108338e360, callargs=callargs@entry=0x7f107e054190, kwdict=kwdict@entry=0x7f107f4ba450, use_tracing=255)
    at ../Python/ceval.c:7351
#54 0x000055d50a4f7fe9 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=0x7f1084d06da0, frame@entry=0x7f1084d06c68, throwflag=throwflag@entry=0) at ../Python/ceval.c:5376
#55 0x000055d50a4fa0b2 in _PyEval_EvalFrame (throwflag=0, frame=0x7f1084d06c68, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#56 _PyEval_Vector (tstate=0x55d50a895a38 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:6429
#57 0x000055d50a3c2ac0 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:393
#58 0x000055d50a3c2521 in _PyVectorcall_Call (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=0x55d50a3c2a6c <_PyFunction_Vectorcall>, callable=callable@entry=0x7f108338dc80, tuple=tuple@entry=0x7f107ef72850, 
    kwargs=kwargs@entry=0x0) at ../Objects/call.c:245
#59 0x000055d50a3c29d5 in _PyObject_Call (tstate=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f108338dc80, args=args@entry=0x7f107ef72850, kwargs=kwargs@entry=0x0) at ../Objects/call.c:328
#60 0x000055d50a3c2a42 in PyObject_Call (callable=callable@entry=0x7f108338dc80, args=args@entry=0x7f107ef72850, kwargs=kwargs@entry=0x0) at ../Objects/call.c:355
#61 0x000055d50a4e3a90 in do_call_core (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=func@entry=0x7f108338dc80, callargs=callargs@entry=0x7f107ef72850, kwdict=kwdict@entry=0x0, use_tracing=255)
    at ../Python/ceval.c:7351
#62 0x000055d50a4f7fe9 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=0x7f1084d06b78, frame@entry=0x7f1084d06a30, throwflag=throwflag@entry=0) at ../Python/ceval.c:5376
#63 0x000055d50a4fa0b2 in _PyEval_EvalFrame (throwflag=0, frame=0x7f1084d06a30, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#64 _PyEval_Vector (tstate=0x55d50a895a38 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:6429
#65 0x000055d50a3c2ac0 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:393
#66 0x000055d50a3c3403 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083b49230, args=args@entry=0x7ffcd8fcd510, nargsf=nargsf@entry=1, 
    kwargs=kwargs@entry=0x7f107f4ba6f0) at ../Objects/call.c:152
#67 0x000055d50a3c35d3 in _PyObject_Call_Prepend (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083b49230, obj=obj@entry=0x7f1083182090, args=args@entry=0x55d50a87b698 <_PyRuntime+58904>, 
    kwargs=kwargs@entry=0x7f107f4ba6f0) at ../Objects/call.c:482
#68 0x000055d50a44ad13 in slot_tp_call (self=0x7f1083182090, args=0x55d50a87b698 <_PyRuntime+58904>, kwds=0x7f107f4ba6f0) at ../Objects/typeobject.c:7630
#69 0x000055d50a3c2c93 in _PyObject_MakeTpCall (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083182090, args=args@entry=0x7f1084d06a18, nargs=<optimized out>, 
    keywords=keywords@entry=0x7f10834999f0) at ../Objects/call.c:214
#70 0x000055d50a3c304f in _PyObject_VectorcallTstate (tstate=0x55d50a895a38 <_PyRuntime+166328>, callable=0x7f1083182090, args=0x7f1084d06a18, nargsf=<optimized out>, kwnames=0x7f10834999f0) at ../Include/internal/pycore_call.h:90
#71 0x000055d50a3c3077 in PyObject_Vectorcall (callable=callable@entry=0x7f1083182090, args=args@entry=0x7f1084d06a18, nargsf=nargsf@entry=9223372036854775808, kwnames=kwnames@entry=0x7f10834999f0) at ../Objects/call.c:299
#72 0x000055d50a4e37b6 in trace_call_function (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=func@entry=0x7f1083182090, args=args@entry=0x7f1084d06a18, nargs=<optimized out>, kwnames=0x7f10834999f0)
    at ../Python/ceval.c:7309
#73 0x000055d50a4f42fc in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=frame@entry=0x7f1084d06998, throwflag=throwflag@entry=0) at ../Python/ceval.c:4766
#74 0x000055d50a4fa0b2 in _PyEval_EvalFrame (throwflag=0, frame=0x7f1084d06998, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#75 _PyEval_Vector (tstate=0x55d50a895a38 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:6429
#76 0x000055d50a3c2ac0 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:393
#77 0x000055d50a3c2521 in _PyVectorcall_Call (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=0x55d50a3c2a6c <_PyFunction_Vectorcall>, callable=callable@entry=0x7f108338f490, tuple=tuple@entry=0x7f1080698be0, 
    kwargs=kwargs@entry=0x0) at ../Objects/call.c:245
#78 0x000055d50a3c29d5 in _PyObject_Call (tstate=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f108338f490, args=args@entry=0x7f1080698be0, kwargs=kwargs@entry=0x0) at ../Objects/call.c:328
#79 0x000055d50a3c2a42 in PyObject_Call (callable=callable@entry=0x7f108338f490, args=args@entry=0x7f1080698be0, kwargs=kwargs@entry=0x0) at ../Objects/call.c:355
#80 0x000055d50a4e3a90 in do_call_core (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=func@entry=0x7f108338f490, callargs=callargs@entry=0x7f1080698be0, kwdict=kwdict@entry=0x0, use_tracing=255)
    at ../Python/ceval.c:7351
#81 0x000055d50a4f7fe9 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=0x7f1084d068a8, frame@entry=0x7f1084d06760, throwflag=throwflag@entry=0) at ../Python/ceval.c:5376
#82 0x000055d50a4fa0b2 in _PyEval_EvalFrame (throwflag=0, frame=0x7f1084d06760, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#83 _PyEval_Vector (tstate=0x55d50a895a38 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:6429
#84 0x000055d50a3c2ac0 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:393
#85 0x000055d50a3c3403 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083b49230, args=args@entry=0x7ffcd8fcdaf0, nargsf=nargsf@entry=1, 
    kwargs=kwargs@entry=0x7f1082304b90) at ../Objects/call.c:152
#86 0x000055d50a3c35d3 in _PyObject_Call_Prepend (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083b49230, obj=obj@entry=0x7f10831824f0, args=args@entry=0x55d50a87b698 <_PyRuntime+58904>, 
    kwargs=kwargs@entry=0x7f1082304b90) at ../Objects/call.c:482
#87 0x000055d50a44ad13 in slot_tp_call (self=0x7f10831824f0, args=0x55d50a87b698 <_PyRuntime+58904>, kwds=0x7f1082304b90) at ../Objects/typeobject.c:7630
#88 0x000055d50a3c2c93 in _PyObject_MakeTpCall (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f10831824f0, args=args@entry=0x7f1084d06758, nargs=<optimized out>, 
    keywords=keywords@entry=0x7f10835f6030) at ../Objects/call.c:214
#89 0x000055d50a3c304f in _PyObject_VectorcallTstate (tstate=0x55d50a895a38 <_PyRuntime+166328>, callable=0x7f10831824f0, args=0x7f1084d06758, nargsf=<optimized out>, kwnames=0x7f10835f6030) at ../Include/internal/pycore_call.h:90
#90 0x000055d50a3c3077 in PyObject_Vectorcall (callable=callable@entry=0x7f10831824f0, args=args@entry=0x7f1084d06758, nargsf=nargsf@entry=9223372036854775808, kwnames=kwnames@entry=0x7f10835f6030) at ../Objects/call.c:299
#91 0x000055d50a4e37b6 in trace_call_function (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=func@entry=0x7f10831824f0, args=args@entry=0x7f1084d06758, nargs=<optimized out>, kwnames=0x7f10835f6030)
    at ../Python/ceval.c:7309
#92 0x000055d50a4f42fc in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=0x7f1084d066f0, frame@entry=0x7f1084d065b8, throwflag=throwflag@entry=0) at ../Python/ceval.c:4766
#93 0x000055d50a4fa0b2 in _PyEval_EvalFrame (throwflag=0, frame=0x7f1084d065b8, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#94 _PyEval_Vector (tstate=0x55d50a895a38 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:6429
#95 0x000055d50a3c2ac0 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:393
#96 0x000055d50a3c2521 in _PyVectorcall_Call (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=0x55d50a3c2a6c <_PyFunction_Vectorcall>, callable=callable@entry=0x7f108338f280, tuple=tuple@entry=0x7f10822711d0, 
    kwargs=kwargs@entry=0x0) at ../Objects/call.c:245
#97 0x000055d50a3c29d5 in _PyObject_Call (tstate=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f108338f280, args=args@entry=0x7f10822711d0, kwargs=kwargs@entry=0x0) at ../Objects/call.c:328
#98 0x000055d50a3c2a42 in PyObject_Call (callable=callable@entry=0x7f108338f280, args=args@entry=0x7f10822711d0, kwargs=kwargs@entry=0x0) at ../Objects/call.c:355
#99 0x000055d50a4e3a90 in do_call_core (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=func@entry=0x7f108338f280, callargs=callargs@entry=0x7f10822711d0, kwdict=kwdict@entry=0x0, use_tracing=0)
    at ../Python/ceval.c:7351
#100 0x000055d50a4f7fe9 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=0x7f1084d064c8, frame@entry=0x7f1084d06380, throwflag=throwflag@entry=0) at ../Python/ceval.c:5376
#101 0x000055d50a4fa0b2 in _PyEval_EvalFrame (throwflag=0, frame=0x7f1084d06380, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#102 _PyEval_Vector (tstate=0x55d50a895a38 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:6429
#103 0x000055d50a3c2ac0 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:393
#104 0x000055d50a3c3403 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083b49230, args=args@entry=0x7ffcd8fce0d0, nargsf=nargsf@entry=1, 
    kwargs=kwargs@entry=0x7f10822f25d0) at ../Objects/call.c:152
#105 0x000055d50a3c35d3 in _PyObject_Call_Prepend (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083b49230, obj=obj@entry=0x7f1083166ef0, args=args@entry=0x55d50a87b698 <_PyRuntime+58904>, 
    kwargs=kwargs@entry=0x7f10822f25d0) at ../Objects/call.c:482
#106 0x000055d50a44ad13 in slot_tp_call (self=0x7f1083166ef0, args=0x55d50a87b698 <_PyRuntime+58904>, kwds=0x7f10822f25d0) at ../Objects/typeobject.c:7630
#107 0x000055d50a3c2c93 in _PyObject_MakeTpCall (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083166ef0, args=args@entry=0x7f1084d06358, nargs=<optimized out>, 
    keywords=keywords@entry=0x7f10835f6a30) at ../Objects/call.c:214
#108 0x000055d50a3c304f in _PyObject_VectorcallTstate (tstate=0x55d50a895a38 <_PyRuntime+166328>, callable=0x7f1083166ef0, args=0x7f1084d06358, nargsf=<optimized out>, kwnames=0x7f10835f6a30) at ../Include/internal/pycore_call.h:90
#109 0x000055d50a3c3077 in PyObject_Vectorcall (callable=callable@entry=0x7f1083166ef0, args=args@entry=0x7f1084d06358, nargsf=<optimized out>, kwnames=kwnames@entry=0x7f10835f6a30) at ../Objects/call.c:299
#110 0x000055d50a4f4558 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=0x7f1084d062a8, frame@entry=0x7f1084d061b8, throwflag=throwflag@entry=0) at ../Python/ceval.c:4771
#111 0x000055d50a4fa0b2 in _PyEval_EvalFrame (throwflag=0, frame=0x7f1084d061b8, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#112 _PyEval_Vector (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=func@entry=0x7f1083fb9bd0, locals=locals@entry=0x7f1083f5b2f0, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0)
    at ../Python/ceval.c:6429
#113 0x000055d50a4fa195 in PyEval_EvalCode (co=co@entry=0x7f10832cb0a0, globals=globals@entry=0x7f1083f5b2f0, locals=locals@entry=0x7f1083f5b2f0) at ../Python/ceval.c:1154
#114 0x000055d50a4d86a3 in builtin_exec_impl (module=module@entry=0x7f108412b770, source=0x7f10832cb0a0, globals=0x7f1083f5b2f0, locals=0x7f1083f5b2f0, closure=0x0) at ../Python/bltinmodule.c:1075
#115 0x000055d50a4d8807 in builtin_exec (module=0x7f108412b770, args=<optimized out>, nargs=2, kwnames=<optimized out>) at ../Python/clinic/bltinmodule.c.h:465
#116 0x000055d50a41fade in cfunction_vectorcall_FASTCALL_KEYWORDS (func=0x7f108412be90, args=0x7f1084d06180, nargsf=<optimized out>, kwnames=0x0) at ../Objects/methodobject.c:443
#117 0x000055d50a3c2f5e in _PyObject_VectorcallTstate (tstate=0x55d50a895a38 <_PyRuntime+166328>, callable=0x7f108412be90, args=0x7f1084d06180, nargsf=9223372036854775810, kwnames=0x0) at ../Include/internal/pycore_call.h:92
#118 0x000055d50a3c3077 in PyObject_Vectorcall (callable=callable@entry=0x7f108412be90, args=args@entry=0x7f1084d06180, nargsf=<optimized out>, kwnames=kwnames@entry=0x0) at ../Objects/call.c:299
#119 0x000055d50a4f4558 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=0x7f1084d060d8, frame@entry=0x7f1084d06020, throwflag=throwflag@entry=0) at ../Python/ceval.c:4771
#120 0x000055d50a4fa0b2 in _PyEval_EvalFrame (throwflag=0, frame=0x7f1084d06020, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#121 _PyEval_Vector (tstate=0x55d50a895a38 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:6429
#122 0x000055d50a3c2ac0 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:393
#123 0x000055d50a3c2521 in _PyVectorcall_Call (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=0x55d50a3c2a6c <_PyFunction_Vectorcall>, callable=callable@entry=0x7f1083e63c20, tuple=tuple@entry=0x7f1083f977a0, 
    kwargs=kwargs@entry=0x0) at ../Objects/call.c:245
#124 0x000055d50a3c29d5 in _PyObject_Call (tstate=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083e63c20, args=args@entry=0x7f1083f977a0, kwargs=kwargs@entry=0x0) at ../Objects/call.c:328
#125 0x000055d50a3c2a42 in PyObject_Call (callable=callable@entry=0x7f1083e63c20, args=args@entry=0x7f1083f977a0, kwargs=kwargs@entry=0x0) at ../Objects/call.c:355
#126 0x000055d50a57332f in pymain_run_module (modname=<optimized out>, set_argv0=set_argv0@entry=1) at ../Modules/main.c:300
#127 0x000055d50a5738b7 in pymain_run_python (exitcode=exitcode@entry=0x7ffcd8fce814) at ../Modules/main.c:595
#128 0x000055d50a57395b in Py_RunMain () at ../Modules/main.c:680
#129 0x000055d50a5739d5 in pymain_main (args=args@entry=0x7ffcd8fce870) at ../Modules/main.c:710
#130 0x000055d50a573aa3 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at ../Modules/main.c:734
#131 0x000055d50a323e06 in main (argc=<optimized out>, argv=<optimized out>) at ../Programs/python.c:15

@brandtbucher
Copy link
Member

brandtbucher commented Sep 23, 2022 •
edited
Loading

But the interesting part is the last part of the traceback, where somehow code becomes a unnitest.mock.Mock. This caused .co_positions to not fail, as it will just return a mock object, but the later usage with itertools did. So, somehow, tb.tb_frame.f_code is a mock (assuming that it is gotten from one of the objects in our test suite), but that doesnt look quite right to me.

Yeah, in general this error is manifesting itself as "code objects are turning into something else". Sometimes that errors in Python land, sometimes it segfaults in C land.

I'm 99% sure that what's happening is some sort of use-after-free. Basically, the refcount of a code object is wrong, so it gets deallocated and its memory is later used for some other, new, object (I've seen sentinels, functions, mocks, and other things). It's tough to identify the root cause, though: it could just be bad refcounting somewhere, or it could be yet another symptom of broken internal state.

And yeah, the common things seem to be that this is caused by resource warnings during GC, while tracing, with generators/coroutines on the stack somewhere above.

One really helpful thing that you could do (since you're way more familiar with the library than I am) would be to try to reproduce this without any (or at least as few as possible) third-party C extensions loaded. It looks like when I'm running $ pytest tests/hikari/impl/test_rest_bot.py (without pytest-randomly or pytest-cov), the following extensions get loaded at some point: multidict._multidict, yarl._quoting_c, aiohttp._helpers, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket, _cffi_backend, and frozenlist._frozenlist. Maybe you could find a way to cut that number down, while still preserving the errors/crashes? Some of them might be pytest's, so if we can find a way to reproduce without using pytest, that would help too.

@brandtbucher
Copy link
Member

Also, for my own notes: this also happens with --assert plain, so pytest's assert rewriting isn't relevant here.

@davfsa
Copy link
Author

davfsa commented Sep 23, 2022 •
edited
Loading

One really helpful thing that you could do (since you're way more familiar with the library than I am) would be to try to reproduce this without any (or at least as few as possible) third-party C extensions loaded. It looks like when I'm running $ pytest tests/hikari/impl/test_rest_bot.py (without pytest-randomly or pytest-cov), the following extensions get loaded at some point: multidict._multidict, yarl._quoting_c, aiohttp._helpers, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket, _cffi_backend, and frozenlist._frozenlist. Maybe you could find a way to cut that number down, while still preserving the errors/crashes? Some of them might be pytest's, so if we can find a way to reproduce without using pytest, that would help too.

That is what I have been trying to do this morning, as the underlying issue for a ResourceWarning is an open socket.socket. But every time the error gets handled nicely and I can't seem to cause a crash. I'll have a try again later today :)

Also, all those modules are not pytest, they are loaded by us. I was trying to find out exactly where the GCed object comes from to find the test that seems to cause this. I'll try figuring that out too

@brandtbucher
Copy link
Member

Disabling the event loop warning in BaseEventLoop.__del__ gets rid of the error, so that seems to confirm my theory.

@brandtbucher
Copy link
Member

So we need a warning or error to happen in __del__ somewhere, with tracing active, while the stack is in some problematic state. Let's try to do that without third-party C extensions.

@davfsa
Copy link
Author

davfsa commented Sep 23, 2022

Disabling the event loop warning in BaseEventLoop.__del__ gets rid of the error, so that seems to confirm my theory.

The stack for the core dump seems to indicate that it happens on a __del__ of an object (in this case so far, socket.socket). I have had runs where editing code and then re-running the test suite causes no errors, bus after two or three runs it does.

Not sure if the BaseEventLoop has much place here. Just wanted to make a comment on that, I am most probably wrong

@brandtbucher
Copy link
Member

brandtbucher commented Sep 23, 2022 •
edited
Loading

Let's start by working on the same reproducer. I'm running pytest tests/hikari/impl/test_rest_bot.py, without pytest-cov or pytest-randomly installed, but with these lines added to the top of the test file:

def trace(frame, event, arg):
    return trace

# sys.settrace(trace)

Running this succeeds (and prints a ResourceWarning):

================================ test session starts =================================
platform linux -- Python 3.11.0rc2, pytest-7.1.3, pluggy-1.0.0
rootdir: /home/brandtbucher/hikari, configfile: pyproject.toml
plugins: cov-3.0.0, asyncio-0.19.0
asyncio: mode=Mode.STRICT
collected 24 items                                                                   

tests/hikari/impl/test_rest_bot.py ........................                    [100%]

================================= 24 passed in 1.60s =================================
/home/brandtbucher/cpython/Lib/asyncio/base_events.py:689: ResourceWarning: unclosed event loop <_UnixSelectorEventLoop running=False closed=False debug=True>
  _warn(f"unclosed event loop {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback

Uncommenting the sys.settrace(trace) line then makes it segfault consistently:

================================ test session starts =================================
platform linux -- Python 3.11.0rc2, pytest-7.1.3, pluggy-1.0.0
rootdir: /home/brandtbucher/hikari, configfile: pyproject.toml
plugins: cov-3.0.0, asyncio-0.19.0
asyncio: mode=Mode.STRICT
collected 24 items                                                                   

tests/hikari/impl/test_rest_bot.py ...............XXX lineno: 41, opcode: 253
Fatal Python error: Segmentation fault

Current thread 0x00007f2accd6d280 (most recent call first):
  Garbage-collecting
  File "/home/brandtbucher/cpython/Lib/ast.py", line 50 in parse
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/_code/source.py", line 185 in getstatementrange_ast
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/_code/code.py", line 260 in getsource
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/_code/code.py", line 718 in _getentrysource
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/_code/code.py", line 810 in repr_traceback_entry
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/_code/code.py", line 867 in repr_traceback
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/_code/code.py", line 926 in repr_excinfo
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/_code/code.py", line 666 in getrepr
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/nodes.py", line 475 in _repr_failure_py
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/python.py", line 1795 in repr_failure
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/reports.py", line 345 in from_item_and_call
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/runner.py", line 365 in pytest_runtest_makereport
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/runner.py", line 221 in call_and_report
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/runner.py", line 130 in runtestprotocol
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/runner.py", line 111 in pytest_runtest_protocol
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/main.py", line 347 in pytest_runtestloop
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/main.py", line 322 in _main
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/main.py", line 268 in wrap_session
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/main.py", line 315 in pytest_cmdline_main
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/config/__init__.py", line 164 in main
  File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/config/__init__.py", line 187 in console_main
  File "/home/brandtbucher/env311/bin/pytest", line 8 in <module>

Extension modules: multidict._multidict, yarl._quoting_c, aiohttp._helpers, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket, _cffi_backend, frozenlist._frozenlist (total: 8)
Segmentation fault (core dumped)

I'd like to have this same file fail (either segfaults or weird Python errors) with as little third-party code as possible (especially C code).

@brandtbucher
Copy link
Member

I'm currently working on bisecting the errors to a specific CPython revision.

@brandtbucher
Copy link
Member

brandtbucher commented Sep 23, 2022 •
edited
Loading

It's not the code objects that are being replaced, it's the frames. I just saw this failure:

AttributeError: 'bool' object has no attribute 'co_positions'

It's not possible for a bool to live where a code object used to. So the frames are being replaced, and the frame->f_frame->f_code move in PyFrame_GetCode is returning a pointer to another object.

It could even be our stack of _PyInterpreterFrames. If ownership of these isn't handled correctly, frame->f_frame could point to some other _PyInterpreterFrame... or even into the middle of one.

(This would also explain the Objects/frameobject.c:1304: PyFrame_GetCode: Assertion `code != NULL' failed failures I saw before.)

@davfsa
Copy link
Author

davfsa commented Sep 24, 2022 •
edited
Loading

Got some more information, not sure how relevant this would be, but posting it just in case.

Simpler reproduction code

This has been tricky, as it seems it is very inconsistent on when it occurs (probably due to gc). Removing/adding code can lead to very different results. Because of this, I havent attempted much to make it smaller, but this is what I am currently using:

Commit: hikari-py/hikari@68ee392

Patch 
diff --git a/tests/hikari/impl/test_rest_bot.py b/tests/hikari/impl/test_rest_bot.py
index ab2120c0..2b6a7b89 100644
--- a/tests/hikari/impl/test_rest_bot.py
+++ b/tests/hikari/impl/test_rest_bot.py
@@ -38,6 +38,17 @@ from hikari.internal import ux
 from tests.hikari import hikari_test_helpers
 
 
+def trace(frame, event, arg):
+    return trace
+
+
+sys.settrace(trace)
+import gc
+
+v = 10
+gc.set_threshold(v, v, v)
+
+
 class TestRESTBot:
     @pytest.fixture()
     def mock_interaction_server(self):
@@ -231,64 +242,19 @@ class TestRESTBot:
         assert mock_rest_bot.is_alive is expected
 
     def test_print_banner(self, mock_rest_bot):
-        with mock.patch.object(ux, "print_banner") as print_banner:
-            mock_rest_bot.print_banner("okokok", True, False, {"test_key": "test_value"})
-
-            print_banner.assert_called_once_with("okokok", True, False, extra_args={"test_key": "test_value"})
+        pass
 
     @pytest.mark.asyncio()
     async def test_close(self, mock_rest_bot, mock_interaction_server, mock_rest_client):
-        mock_rest_bot._close_event = close_event = mock.Mock()
-        mock_interaction_server.close = mock.AsyncMock()
-        mock_rest_bot._is_closing = False
-
-        await mock_rest_bot.close()
-
-        mock_interaction_server.close.assert_awaited_once()
-        mock_rest_client.close.assert_awaited_once()
-        close_event.set.assert_called_once()
-        assert mock_rest_bot._is_closing is True
+        pass
 
     @pytest.mark.asyncio()
     async def test_close_when_is_closing(self, mock_rest_bot, mock_interaction_server, mock_rest_client):
-        mock_rest_bot._close_event = mock.Mock()
-        mock_interaction_server.close = mock.AsyncMock()
-        mock_rest_bot._is_closing = True
-        mock_rest_bot.join = mock.AsyncMock()
-
-        await mock_rest_bot.close()
-
-        mock_interaction_server.close.assert_not_called()
-        mock_rest_client.close.assert_not_called()
-        mock_rest_bot._close_event.set.assert_not_called()
-        mock_rest_bot.join.assert_awaited_once()
+        pass
 
     @pytest.mark.asyncio()
     async def test_close_when_inactive(self, mock_rest_bot):
-        with pytest.raises(errors.ComponentStateConflictError):
-            await mock_rest_bot.close()
-
-    @pytest.mark.asyncio()
-    async def test_join(self, mock_rest_bot):
-        mock_rest_bot._close_event = mock.AsyncMock()
-
-        await mock_rest_bot.join()
-
-        mock_rest_bot._close_event.wait.assert_awaited_once()
-
-    @pytest.mark.asyncio()
-    async def test_join_when_not_alive(self, mock_rest_bot):
-        with pytest.raises(errors.ComponentStateConflictError):
-            await mock_rest_bot.join()
-
-    @pytest.mark.asyncio()
-    async def test_on_interaction(self, mock_rest_bot, mock_interaction_server):
-        mock_interaction_server.on_interaction = mock.AsyncMock()
-
-        result = await mock_rest_bot.on_interaction(b"1", b"2", b"3")
-
-        assert result is mock_interaction_server.on_interaction.return_value
-        mock_interaction_server.on_interaction.assert_awaited_once_with(b"1", b"2", b"3")
+        pass
 
     def test_run(self, mock_rest_bot):
         # Dependent on test-order the current event loop may be pre-set and closed without pytest.mark.asyncio
@@ -345,164 +311,3 @@ class TestRESTBot:
             mock_rest_bot.run(asyncio_debug=True)
 
         get_or_make_loop.return_value.set_debug.assert_called_once_with(True)
-
-    def test_run_with_coroutine_tracking_depth(self, mock_rest_bot):
-        mock_rest_bot.start = mock.Mock()
-        mock_rest_bot.join = mock.Mock()
-
-        with mock.patch.object(aio, "get_or_make_loop"):
-            with mock.patch.object(
-                sys, "set_coroutine_origin_tracking_depth", side_effect=AttributeError, create=True
-            ) as set_tracking_depth:
-                mock_rest_bot.run(coroutine_tracking_depth=42)
-
-        set_tracking_depth.assert_called_once_with(42)
-
-    def test_run_when_already_running(self, mock_rest_bot):
-        mock_rest_bot._close_event = object()
-
-        with pytest.raises(errors.ComponentStateConflictError):
-            mock_rest_bot.run()
-
-    def test_run_closes_executor_when_present(self, mock_rest_bot, mock_executor):
-        mock_rest_bot.join = mock.AsyncMock()
-        # Dependent on test-order the current event loop may be pre-set and closed without pytest.mark.asyncio
-        # therefore we need to ensure there's no pre-set event loop.
-        asyncio.set_event_loop(None)
-        mock_rest_bot.run(
-            asyncio_debug=False,
-            backlog=321,
-            check_for_updates=False,
-            close_loop=False,
-            close_passed_executor=True,
-            coroutine_tracking_depth=32123,
-            enable_signal_handlers=True,
-            host="192.168.1.102",
-            path="pathathath",
-            port=4554,
-            reuse_address=True,
-            reuse_port=False,
-            shutdown_timeout=534.534,
-            socket=object(),
-            ssl_context=object(),
-        )
-
-        mock_executor.shutdown.assert_called_once_with(wait=True)
-        assert mock_rest_bot.executor is None
-
-    def test_run_ignores_close_executor_when_not_present(self, mock_rest_bot):
-        # Dependent on test-order the current event loop may be pre-filled and closed without pytest.mark.asyncio
-        # therefore we need to ensure there's no pre-set event loop.
-        asyncio.set_event_loop(None)
-        mock_rest_bot.join = mock.AsyncMock()
-        mock_rest_bot._executor = None
-
-        mock_rest_bot.run(
-            asyncio_debug=False,
-            backlog=321,
-            check_for_updates=False,
-            close_loop=False,
-            close_passed_executor=True,
-            coroutine_tracking_depth=32123,
-            enable_signal_handlers=True,
-            host="192.168.1.102",
-            path="pathathath",
-            port=4554,
-            reuse_address=True,
-            reuse_port=False,
-            shutdown_timeout=534.534,
-            socket=object(),
-            ssl_context=object(),
-        )
-
-        assert mock_rest_bot.executor is None
-
-    @pytest.mark.asyncio()
-    async def test_start(self, mock_rest_bot, mock_interaction_server, mock_rest_client):
-        mock_socket = object()
-        mock_ssl_context = object()
-        mock_rest_bot._is_closing = True
-
-        with mock.patch.object(ux, "check_for_updates"):
-            await mock_rest_bot.start(
-                backlog=34123,
-                check_for_updates=False,
-                enable_signal_handlers=False,
-                host="hostostosot",
-                port=123123123,
-                path="patpatpapt",
-                reuse_address=True,
-                reuse_port=False,
-                socket=mock_socket,
-                shutdown_timeout=4312312.3132132,
-                ssl_context=mock_ssl_context,
-            )
-
-            ux.check_for_updates.assert_not_called()
-
-        mock_interaction_server.start.assert_awaited_once_with(
-            backlog=34123,
-            enable_signal_handlers=False,
-            host="hostostosot",
-            port=123123123,
-            path="patpatpapt",
-            reuse_address=True,
-            reuse_port=False,
-            socket=mock_socket,
-            shutdown_timeout=4312312.3132132,
-            ssl_context=mock_ssl_context,
-        )
-        mock_rest_client.start.assert_called_once_with()
-        assert mock_rest_bot._is_closing is False
-
-    @pytest.mark.asyncio()
-    async def test_start_checks_for_update(self, mock_rest_bot, mock_http_settings, mock_proxy_settings):
-        stack = contextlib.ExitStack()
-        stack.enter_context(mock.patch.object(asyncio, "create_task"))
-        stack.enter_context(mock.patch.object(ux, "check_for_updates", new=mock.Mock()))
-
-        with stack:
-            await mock_rest_bot.start(
-                backlog=34123,
-                check_for_updates=True,
-                enable_signal_handlers=False,
-                host="hostostosot",
-                port=123123123,
-                path="patpatpapt",
-                reuse_address=True,
-                reuse_port=False,
-                socket=object(),
-                shutdown_timeout=4312312.3132132,
-                ssl_context=object(),
-            )
-
-            asyncio.create_task.assert_called_once_with(
-                ux.check_for_updates.return_value, name="check for package updates"
-            )
-            ux.check_for_updates.assert_called_once_with(mock_http_settings, mock_proxy_settings)
-
-    @pytest.mark.asyncio()
-    async def test_start_when_is_alive(self, mock_rest_bot):
-        mock_rest_bot._close_event = object()
-
-        with mock.patch.object(ux, "check_for_updates", new=mock.Mock()) as check_for_updates:
-            with pytest.raises(errors.ComponentStateConflictError):
-                await mock_rest_bot.start()
-
-            check_for_updates.assert_not_called()
-
-    def test_get_listener(self, mock_rest_bot, mock_interaction_server):
-        mock_type = object()
-
-        result = mock_rest_bot.get_listener(mock_type)
-
-        assert result is mock_interaction_server.get_listener.return_value
-        mock_interaction_server.get_listener.assert_called_once_with(mock_type)
-
-    def test_set_listener(self, mock_rest_bot, mock_interaction_server):
-        mock_type = object()
-        mock_listener = object()
-
-        mock_rest_bot.set_listener(mock_type, mock_listener, replace=True)
-
-        mock_interaction_server.set_listener.assert_called_once_with(mock_type, mock_listener, replace=True)

Removing any more code makes the crash disappear. I havent attempted to lower it even more beyond this. It is also worth noting that I am unable to get 100% crash rate. Sometimes it will just error our with a traceback originated from traceback.py, but crashes can be quite consistent.

I have unfortunately not been able to get rid of the C extensions, as doing so (just like removing/adding code) makes the crash not happen. I do not think this is due to the C extensions causing the crash, but rather the sensitivity of the crash happening.

Crash upon printing or accessing attributes in tb.tb_frame

After modifying _walk_tb_with_full_positions to just drop to a Pdb session on error to try and poke around the tb object, I found that trying to access any attribute on it is what is causing the segfault.

def _walk_tb_with_full_positions(tb):
    # Internal version of walk_tb that yields full code positions including
    # end line and column information.
    while tb is not None:
-       positions = _get_code_position(tb.tb_frame.f_code, tb.tb_lasti)
+       try:
+           positions = _get_code_position(tb.tb_frame.f_code, tb.tb_lasti)
+       except:
+           breakpoint()
        # Yield tb_lineno when co_positions does not have a line number to
        # maintain behavior with walk_tb.
        if positions[0] is None:
            yield tb.tb_frame, (tb.tb_lineno, ) + positions[1:]
        else:
            yield tb.tb_frame, positions
        tb = tb.tb_next
Example 1 (attemting to print it) 
platform linux -- Python 3.11.0rc2+, pytest-7.1.3, pluggy-1.0.0 -- /home/davfsa/coding/hikari/../cpython/debug/python
cachedir: .pytest_cache
rootdir: /home/davfsa/coding/hikari, configfile: pyproject.toml
plugins: asyncio-0.19.0
asyncio: mode=Mode.STRICT
collected 12 items                                                                                                                                                                                                                     

tests/hikari/impl/test_rest_bot.py::TestRESTBot::test___init__ PASSED                                                                                                                                                            [  8%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test___init___parses_string_public_key PASSED                                                                                                                                   [ 16%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test___init___strips_token PASSED                                                                                                                                               [ 25%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test___init___generates_default_settings PASSED                                                                                                                                 [ 33%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test_is_alive_property[<object object at 0x7f5188917580>-True] PASSED                                                                                                           [ 41%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test_is_alive_property[None-False] PASSED                                                                                                                                       [ 50%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test_print_banner PASSED                                                                                                                                                        [ 58%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test_close PASSED                                                                                                                                                               [ 66%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test_close_when_is_closing PASSED                                                                                                                                               [ 75%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test_close_when_inactive PASSED                                                                                                                                                 [ 83%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test_run PASSED                                                                                                                                                                 [ 91%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test_run_when_asyncio_debug 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /home/davfsa/coding/cpython/Lib/traceback.py(394)_walk_tb_with_full_positions()
-> if positions[0] is None:
(Pdb) tb
<traceback object at 0x7f51865158b0>
(Pdb) tb.tb_frame
Fatal Python error: Segmentation fault

Current thread 0x00007f518ad53740 (most recent call first):
  File "/home/davfsa/coding/cpython/Lib/pdb.py", line 444 in displayhook
  File "<stdin>", line 1 in <module>
  File "/home/davfsa/coding/cpython/Lib/pdb.py", line 459 in default
  File "/home/davfsa/coding/cpython/Lib/cmd.py", line 216 in onecmd
  File "/home/davfsa/coding/cpython/Lib/pdb.py", line 500 in onecmd
  File "/home/davfsa/coding/cpython/Lib/cmd.py", line 138 in cmdloop
  File "/home/davfsa/coding/cpython/Lib/pdb.py", line 400 in _cmdloop
  File "/home/davfsa/coding/cpython/Lib/pdb.py", line 435 in interaction
  File "/home/davfsa/coding/cpython/Lib/pdb.py", line 340 in user_line
  File "/home/davfsa/coding/cpython/Lib/bdb.py", line 114 in dispatch_line
  File "/home/davfsa/coding/cpython/Lib/bdb.py", line 90 in trace_dispatch
  File "/home/davfsa/coding/cpython/Lib/traceback.py", line 394 in _walk_tb_with_full_positions
  File "/home/davfsa/coding/cpython/Lib/traceback.py", line 454 in _extract_from_extended_frame_gen
  File "/home/davfsa/coding/cpython/Lib/traceback.py", line 740 in __init__
  File "/home/davfsa/coding/cpython/Lib/traceback.py", line 155 in format_exception
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/unraisableexception.py", line 72 in unraisable_exception_runtest_hook
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/unraisableexception.py", line 88 in pytest_runtest_call
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_callers.py", line 55 in _multicall
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/runner.py", line 259 in <lambda>
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/runner.py", line 338 in from_call
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/runner.py", line 258 in call_runtest_hook
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/runner.py", line 219 in call_and_report
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/runner.py", line 130 in runtestprotocol
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/runner.py", line 111 in pytest_runtest_protocol
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/main.py", line 347 in pytest_runtestloop
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/main.py", line 322 in _main
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/main.py", line 268 in wrap_session
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/main.py", line 315 in pytest_cmdline_main
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/config/__init__.py", line 164 in main
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/config/__init__.py", line 187 in console_main
  File "/home/davfsa/.local/lib/python3.11/site-packages/pytest/__main__.py", line 5 in <module>
  File "/home/davfsa/coding/cpython/Lib/runpy.py", line 88 in _run_code
  File "/home/davfsa/coding/cpython/Lib/runpy.py", line 198 in _run_module_as_main

Extension modules: multidict._multidict, yarl._quoting_c, aiohttp._helpers, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket, frozenlist._frozenlist (total: 7)
Segmentation fault (core dumped)
Example 2 (accessing attributes on it) 
platform linux -- Python 3.11.0rc2+, pytest-7.1.3, pluggy-1.0.0 -- /home/davfsa/coding/hikari/../cpython/debug/python
cachedir: .pytest_cache
rootdir: /home/davfsa/coding/hikari, configfile: pyproject.toml
plugins: asyncio-0.19.0
asyncio: mode=Mode.STRICT
collected 12 items                                                                                                                                                                                                                     

tests/hikari/impl/test_rest_bot.py::TestRESTBot::test___init__ PASSED                                                                                                                                                            [  8%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test___init___parses_string_public_key PASSED                                                                                                                                   [ 16%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test___init___strips_token PASSED                                                                                                                                               [ 25%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test___init___generates_default_settings PASSED                                                                                                                                 [ 33%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test_is_alive_property[<object object at 0x7ff40ad47580>-True] PASSED                                                                                                           [ 41%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test_is_alive_property[None-False] PASSED                                                                                                                                       [ 50%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test_print_banner PASSED                                                                                                                                                        [ 58%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test_close PASSED                                                                                                                                                               [ 66%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test_close_when_is_closing PASSED                                                                                                                                               [ 75%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test_close_when_inactive PASSED                                                                                                                                                 [ 83%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test_run PASSED                                                                                                                                                                 [ 91%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test_run_when_asyncio_debug 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /home/davfsa/coding/cpython/Lib/traceback.py(394)_walk_tb_with_full_positions()
-> if positions[0] is None:
(Pdb) dir(tb.tb_frame)
['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'clear', 'f_back', 'f_builtins', 'f_code', 'f_globals', 'f_lasti', 'f_lineno', 'f_locals', 'f_trace', 'f_trace_lines', 'f_trace_opcodes']
(Pdb) tb.tb_frame.f_code
Fatal Python error: Segmentation fault

Current thread 0x00007ff40d139740 (most recent call first):
  File "<stdin>", line 1 in <module>
  File "/home/davfsa/coding/cpython/Lib/pdb.py", line 459 in default
  File "/home/davfsa/coding/cpython/Lib/cmd.py", line 216 in onecmd
  File "/home/davfsa/coding/cpython/Lib/pdb.py", line 500 in onecmd
  File "/home/davfsa/coding/cpython/Lib/cmd.py", line 138 in cmdloop
  File "/home/davfsa/coding/cpython/Lib/pdb.py", line 400 in _cmdloop
  File "/home/davfsa/coding/cpython/Lib/pdb.py", line 435 in interaction
  File "/home/davfsa/coding/cpython/Lib/pdb.py", line 340 in user_line
  File "/home/davfsa/coding/cpython/Lib/bdb.py", line 114 in dispatch_line
  File "/home/davfsa/coding/cpython/Lib/bdb.py", line 90 in trace_dispatch
  File "/home/davfsa/coding/cpython/Lib/traceback.py", line 394 in _walk_tb_with_full_positions
  File "/home/davfsa/coding/cpython/Lib/traceback.py", line 454 in _extract_from_extended_frame_gen
  File "/home/davfsa/coding/cpython/Lib/traceback.py", line 740 in __init__
  File "/home/davfsa/coding/cpython/Lib/traceback.py", line 155 in format_exception
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/unraisableexception.py", line 72 in unraisable_exception_runtest_hook
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/unraisableexception.py", line 88 in pytest_runtest_call
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_callers.py", line 55 in _multicall
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/runner.py", line 259 in <lambda>
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/runner.py", line 338 in from_call
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/runner.py", line 258 in call_runtest_hook
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/runner.py", line 219 in call_and_report
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/runner.py", line 130 in runtestprotocol
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/runner.py", line 111 in pytest_runtest_protocol
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/main.py", line 347 in pytest_runtestloop
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/main.py", line 322 in _main
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/main.py", line 268 in wrap_session
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/main.py", line 315 in pytest_cmdline_main
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/config/__init__.py", line 164 in main
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/config/__init__.py", line 187 in console_main
  File "/home/davfsa/.local/lib/python3.11/site-packages/pytest/__main__.py", line 5 in <module>
  File "/home/davfsa/coding/cpython/Lib/runpy.py", line 88 in _run_code
  File "/home/davfsa/coding/cpython/Lib/runpy.py", line 198 in _run_module_as_main

Extension modules: multidict._multidict, yarl._quoting_c, aiohttp._helpers, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket, frozenlist._frozenlist (total: 7)
Segmentation fault (core dumped)

Incomplete frames

While looking a the issues that have been mentioned I noticed that they all seem to have incomplete frames as a common factor. I added a simple assertion in PyFrame_GetCode to check if the frame is complete and all the segfaults started to raise because of this (when a traceback wasnt printed, as mentioned above)

PyCodeObject *
PyFrame_GetCode(PyFrameObject *frame)
{
    assert(frame != NULL);
-   PyCodeObject *code = frame->f_frame->f_code;
+   _PyInterpreterFrame *int_frame = frame->f_frame;
+   assert(!_PyFrame_IsIncomplete(int_frame));
+   PyCodeObject *code = int_frame->f_code;
    assert(code != NULL);
    Py_INCREF(code);
    return code;
}
Logs 
platform linux -- Python 3.11.0rc2+, pytest-7.1.3, pluggy-1.0.0 -- /home/davfsa/coding/hikari/../cpython/debug/python
cachedir: .pytest_cache
rootdir: /home/davfsa/coding/hikari, configfile: pyproject.toml
plugins: asyncio-0.19.0
asyncio: mode=Mode.STRICT
collected 12 items                                                                                                                                                                                                                     

tests/hikari/impl/test_rest_bot.py::TestRESTBot::test___init__ PASSED                                                                                                                                                            [  8%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test___init___parses_string_public_key PASSED                                                                                                                                   [ 16%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test___init___strips_token PASSED                                                                                                                                               [ 25%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test___init___generates_default_settings PASSED                                                                                                                                 [ 33%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test_is_alive_property[<object object at 0x7fbb3deb7580>-True] PASSED                                                                                                           [ 41%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test_is_alive_property[None-False] PASSED                                                                                                                                       [ 50%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test_print_banner PASSED                                                                                                                                                        [ 58%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test_close PASSED                                                                                                                                                               [ 66%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test_close_when_is_closing PASSED                                                                                                                                               [ 75%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test_close_when_inactive PASSED                                                                                                                                                 [ 83%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test_run PASSED                                                                                                                                                                 [ 91%]
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test_run_when_asyncio_debug python: ../Objects/frameobject.c:1304: PyFrame_GetCode: Assertion `!_PyFrame_IsIncomplete(frame_obj)' failed.
Fatal Python error: Aborted

Current thread 0x00007fbb402f3740 (most recent call first):
  File "/home/davfsa/coding/cpython/Lib/traceback.py", line 353 in _walk_tb_with_full_positions
  File "/home/davfsa/coding/cpython/Lib/traceback.py", line 416 in _extract_from_extended_frame_gen
  File "/home/davfsa/coding/cpython/Lib/traceback.py", line 688 in __init__
  File "/home/davfsa/coding/cpython/Lib/traceback.py", line 139 in format_exception
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/unraisableexception.py", line 72 in unraisable_exception_runtest_hook
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/unraisableexception.py", line 88 in pytest_runtest_call
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_callers.py", line 55 in _multicall
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/runner.py", line 259 in <lambda>
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/runner.py", line 338 in from_call
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/runner.py", line 258 in call_runtest_hook
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/runner.py", line 219 in call_and_report
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/runner.py", line 130 in runtestprotocol
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/runner.py", line 111 in pytest_runtest_protocol
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/main.py", line 347 in pytest_runtestloop
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/main.py", line 322 in _main
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/main.py", line 268 in wrap_session
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/main.py", line 315 in pytest_cmdline_main
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/davfsa/.local/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/config/__init__.py", line 164 in main
  File "/home/davfsa/.local/lib/python3.11/site-packages/_pytest/config/__init__.py", line 187 in console_main
  File "/home/davfsa/.local/lib/python3.11/site-packages/pytest/__main__.py", line 5 in <module>
  File "/home/davfsa/coding/cpython/Lib/runpy.py", line 88 in _run_code
  File "/home/davfsa/coding/cpython/Lib/runpy.py", line 198 in _run_module_as_main

Extension modules: multidict._multidict, yarl._quoting_c, aiohttp._helpers, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket, frozenlist._frozenlist (total: 7)
Aborted (core dumped)

The CPython version I used here is self-built off 29f17a4 with the only diff being the changes specified in each section)

@brandtbucher
Copy link
Member

I've been working on this for a few days, and have been able to minimize it quite a bit.

Put the following file (crash.py) in a directory all by itself:

import asyncio
import gc
import sys

def trace(frame, event, arg):
    return trace

gc.set_threshold(1, 0, 0)

def test_crasher():
    asyncio.new_event_loop()
    # Allocate bunch of GC-tracked objects. This has the effect of pushing the
    # new event loop (and its soon-to-be-unclosed socket) to the oldest
    # generation and forcing a GC collection shortly after this test completes.
    # The socket will be finalized. When this happens, it raises a warning at a
    # very specific point in pytest's runner machinery, which puts the VM in a
    # bad state and starts breaking stuff:
    [[] for _ in range(6941)]
    # This is needed, perhaps because this bug requires creation of a Python
    # frame object (in this case, as an argument to the trace function)?
    sys.settrace(trace)
    # Old event loop will be collected during the next GC run:
    asyncio.set_event_loop(None)
    # See you on the other side!

Then, from that directory, run:

$ pytest crash.py --verbose --capture no --pythonwarnings error --pythonwarnings ignore::pytest.PytestUnraisableExceptionWarning

All of the options are required to reproduce.

It will hang for a while (about a minute on my machine... the low GC threshold is a real performance-killer), then print the following traceback:

===================================================================================== test session starts =====================================================================================
platform linux -- Python 3.11.0rc2, pytest-7.1.3, pluggy-1.0.0 -- /home/brandtbucher/env311/bin/python
cachedir: .pytest_cache
rootdir: /home/brandtbucher/cpython
collected 1 item                                                                                                                                                                              

crash.py::test_crasher FAILED

========================================================================================== FAILURES ===========================================================================================
________________________________________________________________________________________ test_crasher _________________________________________________________________________________________

exc = <class 'ResourceWarning'>, value = ResourceWarning('unclosed <socket.socket fd=7, family=1, type=1, proto=0>'), tb = <traceback object at 0x7f5f4f349590>, limit = None, chain = True

    def format_exception(exc, /, value=_sentinel, tb=_sentinel, limit=None, \
                         chain=True):
        """Format a stack trace and the exception information.
    
        The arguments have the same meaning as the corresponding arguments
        to print_exception().  The return value is a list of strings, each
        ending in a newline and some containing internal newlines.  When
        these lines are concatenated and printed, exactly the same text is
        printed as does print_exception().
        """
        value, tb = _parse_value_tb(exc, value, tb)
>       te = TracebackException(type(value), value, tb, limit=limit, compact=True)

../Lib/traceback.py:139: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../Lib/traceback.py:688: in __init__
    self.stack = StackSummary._extract_from_extended_frame_gen(
../Lib/traceback.py:416: in _extract_from_extended_frame_gen
    for f, (lineno, end_lineno, colno, end_colno) in frame_gen:
../Lib/traceback.py:353: in _walk_tb_with_full_positions
    positions = _get_code_position(tb.tb_frame.f_code, tb.tb_lasti)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

code = True, instruction_index = 0

    def _get_code_position(code, instruction_index):
        if instruction_index < 0:
            return (None, None, None, None)
>       positions_gen = code.co_positions()
E       AttributeError: 'bool' object has no attribute 'co_positions'

../Lib/traceback.py:366: AttributeError
=================================================================================== short test summary info ===================================================================================
FAILED crash.py::test_crasher - AttributeError: 'bool' object has no attribute 'co_positions'
===================================================================================== 1 failed in 49.22s =====================================================================================

I think asyncio is a red herring. The event loop set/unset code can probably be replaced with something that performs similar allocations and warns in __del__ the same way it does.

The real pain here is pytest itself. There are a lot of very magic frames up above the problematic warning, and at least one of them is giving us trouble.

After modifying _walk_tb_with_full_positions to just drop to a Pdb session on error to try and poke around the tb object, I found that trying to access any attribute on it is what is causing the segfault.

Only the tb.tb_frame member is segfaulting, which leads me to believe that the cached frame object is either (a) not being refcounted correctly or (b) failing to be created, because something about our _PyInterpreterFrame management is broken.

While looking a the issues that have been mentioned I noticed that they all seem to have incomplete frames as a common factor. I added a simple assertion in PyFrame_GetCode to check if the frame is complete and all the segfaults started to raise because of this (when a traceback wasnt printed, as mentioned above)

Yeah, that's odd. No frame object should have an incomplete f_frame member. So either it is being created from an incomplete frame, or the frame is becoming incomplete after the object is created.

@davfsa
Copy link
Author

davfsa commented Sep 28, 2022

Thanks a lot for having a look at this!

Now, I sadly got some bad news. I am unable to reproduce the crash with the code you provided, the test just passes after 1s, but with a small change, I can make it crash in the same way it did for you (takes 33s).

import asyncio
import gc
import sys


def trace(frame, event, arg):
    return trace


gc.set_threshold(1, 0, 0)


def test_crasher():
    asyncio.new_event_loop()
    # Allocate bunch of GC-tracked objects. This has the effect of pushing the
    # new event loop (and its soon-to-be-unclosed socket) to the oldest
    # generation and forcing a GC collection shortly after this test completes.
    # The socket will be finalized. When this happens, it raises a warning at a
    # very specific point in pytest's runner machinery, which puts the VM in a
    # bad state and starts breaking stuff:
-   [[] for _ in range(6941)]
+   a = [[] for _ in range(6941)]
    # This is needed, perhaps because this bug requires creation of a Python
    # frame object (in this case, as an argument to the trace function)?
    sys.settrace(trace)
    # Old event loop will be collected during the next GC run:
    asyncio.set_event_loop(None)
    # See you on the other side!
Logs 
========================================================================================================= test session starts ==========================================================================================================
platform linux -- Python 3.11.0rc2+, pytest-7.1.3, pluggy-1.0.0 -- /home/davfsa/coding/cpython_crash/../cpython/debug/python
cachedir: .pytest_cache
rootdir: /home/davfsa/coding/cpython_crash, configfile: pyproject.toml
plugins: asyncio-0.19.0
asyncio: mode=Mode.STRICT
collected 1 item                                                                                                                                                                                                                       

crash.py::test_crasher PASSED
crash.py::test_crasher ERROR

================================================================================================================ ERRORS ================================================================================================================
__________________________________________________________________________________________________ ERROR at teardown of test_crasher ___________________________________________________________________________________________________

exc = <class 'ResourceWarning'>, value = ResourceWarning('unclosed <socket.socket fd=7, family=1, type=1, proto=0>'), tb = <traceback object at 0x7f56aed5cff0>, limit = None, chain = True

    def format_exception(exc, /, value=_sentinel, tb=_sentinel, limit=None, \
                         chain=True):
        """Format a stack trace and the exception information.
    
        The arguments have the same meaning as the corresponding arguments
        to print_exception().  The return value is a list of strings, each
        ending in a newline and some containing internal newlines.  When
        these lines are concatenated and printed, exactly the same text is
        printed as does print_exception().
        """
        value, tb = _parse_value_tb(exc, value, tb)
>       te = TracebackException(type(value), value, tb, limit=limit, compact=True)

../cpython/Lib/traceback.py:139: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../cpython/Lib/traceback.py:688: in __init__
    self.stack = StackSummary._extract_from_extended_frame_gen(
../cpython/Lib/traceback.py:416: in _extract_from_extended_frame_gen
    for f, (lineno, end_lineno, colno, end_colno) in frame_gen:
../cpython/Lib/traceback.py:353: in _walk_tb_with_full_positions
    positions = _get_code_position(tb.tb_frame.f_code, tb.tb_lasti)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

code = True, instruction_index = 0

    def _get_code_position(code, instruction_index):
        if instruction_index < 0:
            return (None, None, None, None)
>       positions_gen = code.co_positions()
E       AttributeError: 'bool' object has no attribute 'co_positions'

../cpython/Lib/traceback.py:366: AttributeError
======================================================================================================= short test summary info ========================================================================================================
ERROR crash.py::test_crasher - AttributeError: 'bool' object has no attribute 'co_positions'
===================================================================================================== 1 passed, 1 error in 33.15s ======================================================================================================

What i find funny is the initial passing and then failing of the test. @brandtbucher would you mind testing out my diff and seeing if that makes it crash for you too? Then we will both finally have a consistent reproduction (yay!). Again, thank you so much for making it this easy to reproduce, I was unable to go any further.

I am going to make a little script to go back in the git history and try to find the first commit where this doesnt crash, which will hopefully narrow this down even further.

@brandtbucher
Copy link
Member

I think I might be going mad. I have been going around in circles for the past hour. The code that has been causing crashes all night (I left the script to find the problematic commit running over-night) no longer does, no matter what I do to it (this is the code that I posted above). In any commit.

I understand your frustration. That's the nature of these sorts of bugs that rely on precise GC collections: tiny, tiny changes in allocation behavior can make all the difference. Something as simple as new environment variables, import paths, PYC files, etc. can change everything... especially with all the gunk that Pytest runs before the test even begins, and after it ends. I've found that recompiling the same commit multiple times can sometimes require tweaking the magic 6941 value by up to +/-40! My trick is to add a line that prints GC stats to the end of the function, raise the value until it just barely triggers a full GC run (of the oldest generation), then adjust the value downwards until it crashes again.

That value is super fragile, so it probably doesn't lend itself to bisecting to a dirty commit very well. I'm going to try digging into this by figuring out where in the code those incomplete frames are coming from.

@davfsa
Copy link
Author

davfsa commented Sep 30, 2022

Ok, that makes a lot more sense. I had the false sense that you could consistently reproduce it and I was so confused on why I couldn't. Tysm for the insights and how go get the magic number. I'll try and keep reproducing the error today 😄

I don't mind how tedious this bug is, it's something I was well aware when trying to help out, it's more what I described above that I didn't really understand. Thanks for the explanation and your work!

@brandtbucher
Copy link
Member

Here's one with no other Python code (besides, of course, pytest):

# pytest crash.py --verbose --capture no --pythonwarnings error --pythonwarnings ignore::pytest.PytestUnraisableExceptionWarning

import gc
import _socket
import sys

def trace(frame, event, arg):
    return trace

gc.set_threshold(1, 0, 0)

class Socket(_socket.socket):
    pass

class Crasher:
    def __init__(self):
        self.resource = Socket()
        self.cycle = self

def test_crasher():
    Crasher()
    [[] for _ in range(6267)]
    sys.settrace(trace)
    # print("", *gc.get_stats(), gc.get_count(), sep="\n--> ")

Next steps are figuring out what sort of frames pytest is running above us. They're doing something that we aren't handling correctly, and I want to know what.

@brandtbucher
Copy link
Member

brandtbucher commented Sep 30, 2022 •
edited
Loading

My initial suspicion is that their pattern of hooks-that-are-context-managers-that-are-really-generators could be a big part of this. Pytest's extensive use of this pattern seems like a pretty good stress test of our _PyInterpreterFrame management and ownership rules.

Their --debug option seems useful.

@davfsa
Copy link
Author

davfsa commented Sep 30, 2022

Thanks for the second repo!

I was having a look through pytests code and their unraisable_exception_runtest_hook, which makes use of a context manager with patches sys.unraisablehook might be a nice place to start looking, as the segfaults are raised from there.

I haven't had enough time to have a proper look, but might be useful for you :)

@brandtbucher

@brandtbucher
Copy link
Member

Yep, that caught my eye too! Looking at it now.

@brandtbucher
Copy link
Member

brandtbucher commented Sep 30, 2022 •
edited
Loading

I think I may have found the issue. Frame objects aren't being tracked by the GC when they're created - they're only tracked once they outlive the actual call or generator that they reference. I believe that this is incorrect (it looks like frame objects are smart enough already to only traverse data that they "own").

I'm testing out a fix now to see if I can still reproduce the issue.

False alarm, still crashes. :(

@bedevere-bot bedevere-bot mentioned this issue Oct 6, 2022
Merged
@brandtbucher brandtbucher self-assigned this Oct 6, 2022
@brandtbucher brandtbucher added the 3.12 bugs and security fixes label Oct 6, 2022
@brandtbucher brandtbucher changed the title Attribute error caused by _get_code_position Multiple PyFrameObjects can be created for a single _PyInterpreterFrame Oct 6, 2022
@ezio-melotti ezio-melotti moved this to Todo in Sprint 2024 Oct 6, 2022
@brandtbucher
Copy link
Member

brandtbucher commented Oct 6, 2022 •
edited
Loading

Update: We've been working on this at the CPython core dev sprint at Google this week. I finally found the root issue, which is that a poorly-timed GC collection just before tracing the resumption of a generator can cause two duplicate frame objects to be created, which corrupts the VM's internal state in all sorts of fun ways. Thanks for your help reporting and debugging this!

markshannon pushed a commit that referenced this issue Oct 6, 2022
@brandtbucher
GH-97002: Prevent _PyInterpreterFrames from backing more than one `…
21a2d9f 
…PyFrameObject` (GH-97996)
@bedevere-bot bedevere-bot mentioned this issue Oct 6, 2022
Merged
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Oct 6, 2022
@brandtbucher @miss-islington
pythonGH-97002: Prevent _PyInterpreterFrames from backing more than…
1b2ea2e 
… one `PyFrameObject` (pythonGH-97996)

(cherry picked from commit 21a2d9f)

Co-authored-by: Brandt Bucher <brandtbucher@microsoft.com>
@brandtbucher
Copy link
Member

Confirmed that this fixes the segfaults in the Hikari test suite. This fix will ship in 3.11.0!

brandtbucher pushed a commit that referenced this issue Oct 7, 2022
@miss-islington
[3.11] GH-97002: Prevent _PyInterpreterFrames from backing more than …
89e9474 
…one PyFrameObject (GH-98002)

(cherry picked from commit 21a2d9f)
@davfsa
Copy link
Author

davfsa commented Oct 7, 2022

Confirmed that this fixes the segfaults in the Hikari test suite. This fix will ship in 3.11.0!

Thank you for finding the root cause and fixing it! 🥳

@arhadthedev
Copy link
Member

@brandtbucher Since gh-98002 is merged, can this issue be closed or have its release-blocker label removed?

@brandtbucher
Copy link
Member

I'd prefer to leave it open, and let @pablogsal close it once the cherry-pick to 3.11.0 is finished.

carljm added a commit to carljm/cpython that referenced this issue Oct 8, 2022
@carljm
Merge branch 'main' into typewatch
415ed49 
* main:
  pythonGH-97002: Prevent `_PyInterpreterFrame`s from backing more than one `PyFrameObject` (pythonGH-97996)
  pythongh-97973: Return all necessary information from the tokenizer (pythonGH-97984)
  fixes pythongh-96078: os.sched_yield release the GIL while calling sched_yield(2). (pythongh-97965)
  pythongh-65961: Do not rely solely on `__cached__` (pythonGH-97990)
  pythongh-97850: Remove the open issues section from the import reference (python#97935)
  Docs: pin sphinx-lint (pythonGH-97992)
  pythongh-94590: add signatures to operator itemgetter, attrgetter, methodcaller (python#94591)
  Add Pynche's move to the What's new in 3.11 (python#97974)
  pythongh-97781: Apply changes from importlib_metadata 5. (pythonGH-97785)
  pythongh-86482: Document assignment expression need for ()s (python#23291)
  pythongh-97943: PyFunction_GetAnnotations should return a borrowed reference. (python#97949)
mpage pushed a commit to mpage/cpython that referenced this issue Oct 11, 2022
@brandtbucher @mpage
pythonGH-97002: Prevent _PyInterpreterFrames from backing more than…
cea3697 
… one `PyFrameObject` (pythonGH-97996)
Repository owner moved this from Todo to Done in Release and Deferred blockers 🚫 Oct 22, 2022
Repository owner moved this from Todo to Done in Sprint 2024 Oct 22, 2022
pablogsal pushed a commit that referenced this issue Oct 22, 2022
@miss-islington @pablogsal
[3.11] GH-97002: Prevent _PyInterpreterFrames from backing more than …
d0ab10f 
…one PyFrameObject (GH-98002)

(cherry picked from commit 21a2d9f)
@dmbaggett dmbaggett mentioned this issue Sep 7, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@brandtbucher brandtbucher

Labels
3.11 only security fixes 3.12 bugs and security fixes release-blocker sprint type-crash A hard crash of the interpreter, possibly with a core dump
Projects
Release and Deferred blockers 🚫
Archived in project
Sprint 2024
Archived in project
Milestone
No milestone
Development

No branches or pull requests
4 participants
@arhadthedev @pablogsal @davfsa @brandtbucher