Recording Pypy Segfaults in Testing #2936

CoolCat467 · 2024-01-24T22:43:48Z

Gah, well see the segfaults don't exactly provide much information... But I agree that it's annoying. At this point ig we might as well make an issue here (on trio) to track things cause upstream pytest hasn't been able to solve this yet.

Originally posted by @A5rocks in #2930 (comment)

This issue is for recording segfaults during pypy runs in the github actions testing runner.

So far (also feel free to edit and add more):
Windows x64 pypy 3.9 nightly:
https://github.com/python-trio/trio/actions/runs/5705575738/job/15460400267?pr=2724
Ubuntu pypy-3.10-nightly:
https://github.com/python-trio/trio/actions/runs/7326018868/job/19951219901#step:5:1197
Ubuntu pypy-3.9:
https://github.com/python-trio/trio/actions/runs/7607032357/job/20713702030#step:5:1186
Ubuntu pypy-3.9-nightly:
https://github.com/python-trio/trio/actions/runs/7646760307/job/20836270507#step:5:1188

Not segfault but another error during Ubuntu pypy-3.9:
TypeError: expected some sort of stmt, got '@py_assert1'
https://github.com/python-trio/trio/actions/runs/7054572593/job/19203654087#step:5:1147

The text was updated successfully, but these errors were encountered:

jakkdl · 2024-01-29T15:20:52Z

another instance of the ubuntu pypy-3.9: https://github.com/python-trio/trio/actions/runs/7698138955/job/20976753073#step:5:1193

A5rocks · 2024-01-29T15:53:45Z

Are we sure the Windows segfault is this?

A5rocks · 2024-01-29T23:27:28Z

and another ubuntu pypy-3.9 one: https://github.com/python-trio/trio/actions/runs/7703752769/job/20994747752#step:5:1186

CoolCat467 · 2024-01-30T03:35:11Z

Are we sure the Windows segfault is this?

No, the logs expired for that one.
EDIT:
Issue comment where I mentioned this particular segfault: #2724 (comment)

jakkdl · 2024-01-30T14:34:41Z

https://github.com/python-trio/trio/actions/runs/7712303460/job/21019559589?pr=2933

I think this is just a random network error?

macOS pypy-3.9-nightly

Run codecov/codecov-action@v3
  with:
    directory: empty
    token: 87cefb17-c44b-4f2f-8b30-1fff5769ce46
    name: macOS (pypy-3.9-nightly)
    flags: macOS,pypy-3.9-nightly
==> macos OS detected
Error: getaddrinfo ENOTFOUND uploader.codecov.io
    at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:118:26)

jakkdl · 2024-01-31T13:45:04Z

ubuntu pypy-3.9
https://github.com/python-trio/trio/actions/runs/7725935804/job/21061210801?pr=2941#step:5:1489

  Fatal Python error: Segmentation fault
  
  Stack (most recent call first, approximate line numbers):
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/jedi/api/completion.py", line 68 in filter_names
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/jedi/api/completion.py", line 141 in complete
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/jedi/api/__init__.py", line 194 in complete
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/jedi/api/helpers.py", line 469 in wrapper
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/trio/_tests/test_exports.py", line 245 in test_static_tool_sees_class_members
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/python.py", line 187 in pytest_pyfunc_call
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 27 in _multicall
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_manager.py", line 106 in _hookexec
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 479 in __call__
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/python.py", line 1790 in runtest
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 160 in pytest_runtest_call
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 27 in _multicall
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_manager.py", line 106 in _hookexec
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 479 in __call__
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 262 in <lambda>
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 318 in from_call
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 247 in call_runtest_hook
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 219 in call_and_report
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 119 in runtestprotocol
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 111 in pytest_runtest_protocol
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 27 in _multicall
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_manager.py", line 106 in _hookexec
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 479 in __call__
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 338 in pytest_runtestloop
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 27 in _multicall
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_manager.py", line 106 in _hookexec
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 479 in __call__
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 321 in _main
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 258 in wrap_session
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 27 in _multicall
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_manager.py", line 106 in _hookexec
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 479 in __call__
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/config/__init__.py", line 135 in main
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/config/__init__.py", line 185 in console_main
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pytest/__main__.py", line 1 in <module>
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/coverage/execfile.py", line 169 in run
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/coverage/cmdline.py", line 810 in do_run
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/coverage/cmdline.py", line 608 in command_line
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/coverage/cmdline.py", line 961 in main
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/bin/coverage", line 3 in <module>
    File "<builtin>/app_main.py", line 1017 in execfile
    File "<builtin>/app_main.py", line 131 in run_toplevel
    File "<builtin>/app_main.py", line 737 in run_command_line
    File "<builtin>/app_main.py", line 1131 in entry_point
  ../../../../../../opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/trio/_tests/test_exports.py::test_static_tool_sees_class_members[jedi-trio.lowlevel] ::endgroup::
  ./ci.sh: line 151:  1981 Segmentation fault      (core dumped) COVERAGE_PROCESS_START=$(pwd)/../pyproject.toml coverage run --rcfile=../pyproject.toml -m pytest -ra --junitxml=../test-results.xml --run-slow ${INSTALLDIR} --verbose --durations=10 $flags

A5rocks · 2024-02-05T16:24:55Z

Oh I see I forgot to skip jedi on pypy for test_static_tool_sees_class_members (I fixed that for test_static_tool_sees_all_symbols)

jakkdl · 2024-02-13T14:27:47Z

another instance of the same ubuntu pypy-3.9 error: https://github.com/python-trio/trio/actions/runs/7886026350/job/21518402835#step:5:1142

succeeded on rerun. Looking closer at the logs, the error is from a runner using Runner image version 20240204.1.0, and the success one uses 20240212.1.0. Although 7 hours ago a 20240204.1.0 ubuntu pypy-3.9 run did succeed: https://github.com/python-trio/trio/actions/runs/7883080425/job/21509366109 so that's probably not a thing either.

Zac-HD · 2024-03-28T19:08:56Z

Recently fixed upstream, so I think we're done! pytest-dev/pytest#11168 (comment)

https://www.pypy.org/posts/2024/03/fixing-bug-incremental-gc.html is well worth reading, too.

jakkdl · 2024-03-31T10:06:49Z

Great writeup! Very impressive debugging skillz

CoolCat467 · 2024-04-06T13:19:27Z

We had another one in #2972, https://github.com/python-trio/trio/actions/runs/8579945946/job/23515500181#step:5:1232, have we updated pypy or anything?

Zac-HD · 2024-04-06T19:43:36Z

Looks like the latest version is PyPy 7.3.15, from Jan 15th - we're running this, and the fix just hasn't been released yet. From the post:

At the point of publishing this post, the fixes got merged to the 2.7/3.9/3.10 branches of PyPy, and will be part of the next release (v7.3.16).

Happy to leave this open until PyPy is released, but since we get patch updates automatically via GitHub actions I think we're basically done here.

CoolCat467 · 2024-05-30T17:44:40Z

Looking at https://github.com/pypy/pypy/tags, it looks like v7.3.16 was released on Apr 20, anything we need to do on our end to update pypy version or are we good to close this now?

Zac-HD · 2024-05-30T18:39:46Z

I think we're good to close! Thanks for tracking 😁

Zac-HD closed this as completed Mar 28, 2024

CoolCat467 reopened this Apr 6, 2024

Zac-HD closed this as completed May 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Recording Pypy Segfaults in Testing #2936

Recording Pypy Segfaults in Testing #2936

CoolCat467 commented Jan 24, 2024 •

edited

Loading

jakkdl commented Jan 29, 2024 •

edited by CoolCat467

Loading

A5rocks commented Jan 29, 2024

A5rocks commented Jan 29, 2024 •

edited by CoolCat467

Loading

CoolCat467 commented Jan 30, 2024 •

edited

Loading

jakkdl commented Jan 30, 2024

jakkdl commented Jan 31, 2024

A5rocks commented Feb 5, 2024

jakkdl commented Feb 13, 2024

Zac-HD commented Mar 28, 2024

jakkdl commented Mar 31, 2024

CoolCat467 commented Apr 6, 2024

Zac-HD commented Apr 6, 2024

CoolCat467 commented May 30, 2024

Zac-HD commented May 30, 2024

Recording Pypy Segfaults in Testing #2936

Recording Pypy Segfaults in Testing #2936

Comments

CoolCat467 commented Jan 24, 2024 • edited Loading

jakkdl commented Jan 29, 2024 • edited by CoolCat467 Loading

A5rocks commented Jan 29, 2024

A5rocks commented Jan 29, 2024 • edited by CoolCat467 Loading

CoolCat467 commented Jan 30, 2024 • edited Loading

jakkdl commented Jan 30, 2024

jakkdl commented Jan 31, 2024

A5rocks commented Feb 5, 2024

jakkdl commented Feb 13, 2024

Zac-HD commented Mar 28, 2024

jakkdl commented Mar 31, 2024

CoolCat467 commented Apr 6, 2024

Zac-HD commented Apr 6, 2024

CoolCat467 commented May 30, 2024

Zac-HD commented May 30, 2024

CoolCat467 commented Jan 24, 2024 •

edited

Loading

jakkdl commented Jan 29, 2024 •

edited by CoolCat467

Loading

A5rocks commented Jan 29, 2024 •

edited by CoolCat467

Loading

CoolCat467 commented Jan 30, 2024 •

edited

Loading