[CI] Intermittent SEGV in cmdlineargs tests with threads #46537
Labels
Comments
vtjnash
changed the title
vtjnash
added a commit
that referenced
this issue
Jan 11, 2023
Closes #47590 (pthread_cancel still forbidden though, since async mode will corrupt the process, and synchronously tested is just a slow implementation of a boolean) Refs #47201 (only deals with thread exit, not other case where this is an issue, like cfunction exit and gc-safe-leave) May help #46537, by blocking jl_wake_libuv before uv_library_shutdown, and other tweaks to GC mode. For example: [4011824] signal (6.-6): Aborted gsignal at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) uv__async_send at /workspace/srcdir/libuv/src/unix/async.c:198 uv_async_send at /workspace/srcdir/libuv/src/unix/async.c:73 jl_wake_libuv at /data/vtjnash/julia1/src/jl_uv.c:44 [inlined] JL_UV_LOCK at /data/vtjnash/julia1/src/jl_uv.c:64 [inlined] ijl_iolock_begin at /data/vtjnash/julia1/src/jl_uv.c:72 iolock_begin at ./libuv.jl:48 [inlined] _trywait at ./asyncevent.jl:140 wait at ./asyncevent.jl:155 [inlined] profile_printing_listener at /data/vtjnash/julia1/usr/share/julia/stdlib/v1.10/Profile/src/Profile.jl:39 jfptr_YY.3_58617 at /data/vtjnash/julia1/usr/lib/julia/sys.so (unknown line) _jl_invoke at /data/vtjnash/julia1/src/gf.c:2665 [inlined] ijl_apply_generic at /data/vtjnash/julia1/src/gf.c:2866 jl_apply at /data/vtjnash/julia1/src/julia.h:1870 [inlined] start_task at /data/vtjnash/julia1/src/task.c:1093 Aborted Fixes #37400
vtjnash
added a commit
that referenced
this issue
Jan 11, 2023
Closes #47590 (pthread_cancel still forbidden though, since async mode will corrupt the process, and synchronously tested is just a slow implementation of a boolean) Refs #47201 (only deals with thread exit, not other case where this is an issue, like cfunction exit and gc-safe-leave) May help #46537, by blocking jl_wake_libuv before uv_library_shutdown, and other tweaks to GC mode. For example: [4011824] signal (6.-6): Aborted gsignal at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) uv__async_send at /workspace/srcdir/libuv/src/unix/async.c:198 uv_async_send at /workspace/srcdir/libuv/src/unix/async.c:73 jl_wake_libuv at /data/vtjnash/julia1/src/jl_uv.c:44 [inlined] JL_UV_LOCK at /data/vtjnash/julia1/src/jl_uv.c:64 [inlined] ijl_iolock_begin at /data/vtjnash/julia1/src/jl_uv.c:72 iolock_begin at ./libuv.jl:48 [inlined] _trywait at ./asyncevent.jl:140 wait at ./asyncevent.jl:155 [inlined] profile_printing_listener at /data/vtjnash/julia1/usr/share/julia/stdlib/v1.10/Profile/src/Profile.jl:39 jfptr_YY.3_58617 at /data/vtjnash/julia1/usr/lib/julia/sys.so (unknown line) _jl_invoke at /data/vtjnash/julia1/src/gf.c:2665 [inlined] ijl_apply_generic at /data/vtjnash/julia1/src/gf.c:2866 jl_apply at /data/vtjnash/julia1/src/julia.h:1870 [inlined] start_task at /data/vtjnash/julia1/src/task.c:1093 Aborted Fixes #37400
vtjnash
added a commit
that referenced
this issue
Jan 11, 2023
Closes #47590 (pthread_cancel still forbidden though, since async mode will corrupt the process, and synchronously tested is just a slow implementation of a boolean) Refs #47201 (only deals with thread exit, not other case where this is an issue, like cfunction exit and gc-safe-leave) May help #46537, by blocking jl_wake_libuv before uv_library_shutdown, and other tweaks to GC mode. For example: [4011824] signal (6.-6): Aborted gsignal at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) uv__async_send at /workspace/srcdir/libuv/src/unix/async.c:198 uv_async_send at /workspace/srcdir/libuv/src/unix/async.c:73 jl_wake_libuv at /data/vtjnash/julia1/src/jl_uv.c:44 [inlined] JL_UV_LOCK at /data/vtjnash/julia1/src/jl_uv.c:64 [inlined] ijl_iolock_begin at /data/vtjnash/julia1/src/jl_uv.c:72 iolock_begin at ./libuv.jl:48 [inlined] _trywait at ./asyncevent.jl:140 wait at ./asyncevent.jl:155 [inlined] profile_printing_listener at /data/vtjnash/julia1/usr/share/julia/stdlib/v1.10/Profile/src/Profile.jl:39 jfptr_YY.3_58617 at /data/vtjnash/julia1/usr/lib/julia/sys.so (unknown line) _jl_invoke at /data/vtjnash/julia1/src/gf.c:2665 [inlined] ijl_apply_generic at /data/vtjnash/julia1/src/gf.c:2866 jl_apply at /data/vtjnash/julia1/src/julia.h:1870 [inlined] start_task at /data/vtjnash/julia1/src/task.c:1093 Aborted Fixes #37400
vtjnash
added a commit
that referenced
this issue
Jan 11, 2023
Closes #47590 (pthread_cancel still forbidden though, since async mode will corrupt the process, and synchronously tested is just a slow implementation of a boolean) Refs #47201 (only deals with thread exit, not other case where this is an issue, like cfunction exit and gc-safe-leave) May help #46537, by blocking jl_wake_libuv before uv_library_shutdown, and other tweaks to GC mode. For example: [4011824] signal (6.-6): Aborted gsignal at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) uv__async_send at /workspace/srcdir/libuv/src/unix/async.c:198 uv_async_send at /workspace/srcdir/libuv/src/unix/async.c:73 jl_wake_libuv at /data/vtjnash/julia1/src/jl_uv.c:44 [inlined] JL_UV_LOCK at /data/vtjnash/julia1/src/jl_uv.c:64 [inlined] ijl_iolock_begin at /data/vtjnash/julia1/src/jl_uv.c:72 iolock_begin at ./libuv.jl:48 [inlined] _trywait at ./asyncevent.jl:140 wait at ./asyncevent.jl:155 [inlined] profile_printing_listener at /data/vtjnash/julia1/usr/share/julia/stdlib/v1.10/Profile/src/Profile.jl:39 jfptr_YY.3_58617 at /data/vtjnash/julia1/usr/lib/julia/sys.so (unknown line) _jl_invoke at /data/vtjnash/julia1/src/gf.c:2665 [inlined] ijl_apply_generic at /data/vtjnash/julia1/src/gf.c:2866 jl_apply at /data/vtjnash/julia1/src/julia.h:1870 [inlined] start_task at /data/vtjnash/julia1/src/task.c:1093 Aborted Fixes #37400
vtjnash
added a commit
that referenced
this issue
Jan 13, 2023
Closes #47590 (pthread_cancel still forbidden though, since async mode will corrupt or deadlock the process, and synchronously tested with cancelation disabled whenever this is a lock is just a slow implementation of a boolean) Refs #47201 (only deals with thread exit, not other case where this is an issue, like cfunction exit and gc-safe-leave) May help #46537, by blocking jl_wake_libuv before uv_library_shutdown, and other tweaks to GC mode. For example, avoiding: [4011824] signal (6.-6): Aborted gsignal at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) uv__async_send at /workspace/srcdir/libuv/src/unix/async.c:198 uv_async_send at /workspace/srcdir/libuv/src/unix/async.c:73 jl_wake_libuv at /data/vtjnash/julia1/src/jl_uv.c:44 [inlined] JL_UV_LOCK at /data/vtjnash/julia1/src/jl_uv.c:64 [inlined] ijl_iolock_begin at /data/vtjnash/julia1/src/jl_uv.c:72 iolock_begin at ./libuv.jl:48 [inlined] _trywait at ./asyncevent.jl:140 wait at ./asyncevent.jl:155 [inlined] profile_printing_listener at /data/vtjnash/julia1/usr/share/julia/stdlib/v1.10/Profile/src/Profile.jl:39 jfptr_YY.3_58617 at /data/vtjnash/julia1/usr/lib/julia/sys.so (unknown line) _jl_invoke at /data/vtjnash/julia1/src/gf.c:2665 [inlined] ijl_apply_generic at /data/vtjnash/julia1/src/gf.c:2866 jl_apply at /data/vtjnash/julia1/src/julia.h:1870 [inlined] start_task at /data/vtjnash/julia1/src/task.c:1093 Aborted Fixes #37400
vtjnash
added a commit
that referenced
this issue
Jan 13, 2023
Closes #47590 (pthread_cancel still forbidden though, since async mode will corrupt or deadlock the process, and synchronously tested with cancelation disabled whenever this is a lock is just a slow implementation of a boolean) Refs #47201 (only deals with thread exit, not other case where this is an issue, like cfunction exit and gc-safe-leave) May help #46537, by blocking jl_wake_libuv before uv_library_shutdown, and other tweaks to GC mode. For example, avoiding: [4011824] signal (6.-6): Aborted gsignal at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) uv__async_send at /workspace/srcdir/libuv/src/unix/async.c:198 uv_async_send at /workspace/srcdir/libuv/src/unix/async.c:73 jl_wake_libuv at /data/vtjnash/julia1/src/jl_uv.c:44 [inlined] JL_UV_LOCK at /data/vtjnash/julia1/src/jl_uv.c:64 [inlined] ijl_iolock_begin at /data/vtjnash/julia1/src/jl_uv.c:72 iolock_begin at ./libuv.jl:48 [inlined] _trywait at ./asyncevent.jl:140 wait at ./asyncevent.jl:155 [inlined] profile_printing_listener at /data/vtjnash/julia1/usr/share/julia/stdlib/v1.10/Profile/src/Profile.jl:39 jfptr_YY.3_58617 at /data/vtjnash/julia1/usr/lib/julia/sys.so (unknown line) _jl_invoke at /data/vtjnash/julia1/src/gf.c:2665 [inlined] ijl_apply_generic at /data/vtjnash/julia1/src/gf.c:2866 jl_apply at /data/vtjnash/julia1/src/julia.h:1870 [inlined] start_task at /data/vtjnash/julia1/src/task.c:1093 Aborted Fixes #37400
vtjnash
added a commit
that referenced
this issue
Jan 14, 2023
Closes #47590 (pthread_cancel still forbidden though, since async mode will corrupt or deadlock the process, and synchronously tested with cancelation disabled whenever this is a lock is just a slow implementation of a boolean) Refs #47201 (only deals with thread exit, not other case where this is an issue, like cfunction exit and gc-safe-leave) May help #46537, by blocking jl_wake_libuv before uv_library_shutdown, and other tweaks to GC mode. For example, avoiding: [4011824] signal (6.-6): Aborted gsignal at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) uv__async_send at /workspace/srcdir/libuv/src/unix/async.c:198 uv_async_send at /workspace/srcdir/libuv/src/unix/async.c:73 jl_wake_libuv at /data/vtjnash/julia1/src/jl_uv.c:44 [inlined] JL_UV_LOCK at /data/vtjnash/julia1/src/jl_uv.c:64 [inlined] ijl_iolock_begin at /data/vtjnash/julia1/src/jl_uv.c:72 iolock_begin at ./libuv.jl:48 [inlined] _trywait at ./asyncevent.jl:140 wait at ./asyncevent.jl:155 [inlined] profile_printing_listener at /data/vtjnash/julia1/usr/share/julia/stdlib/v1.10/Profile/src/Profile.jl:39 jfptr_YY.3_58617 at /data/vtjnash/julia1/usr/lib/julia/sys.so (unknown line) _jl_invoke at /data/vtjnash/julia1/src/gf.c:2665 [inlined] ijl_apply_generic at /data/vtjnash/julia1/src/gf.c:2866 jl_apply at /data/vtjnash/julia1/src/julia.h:1870 [inlined] start_task at /data/vtjnash/julia1/src/task.c:1093 Aborted Fixes #37400
|
Similar, but a SIGABRT in cmdlineargs tests at 428d242. Reproducible with repeat details |
|
This is now a persistent, reproducible error on macos ARM starting from 428d242 and continuing to all subsequent commits. If the test in question, line 723 and following in cmdlineargs.jl is run independently in the REPL of the newly built julia, it passes. See end of details section below. details from build and testall of commit 1bfdf98755 details |
|
cmdlineargs error above was present through 113c2f3. As of 12d329b, it disappeared. When I have time, I'll do a bisect between these two. |
|
Close? |
|
Yes, would close for now. This is a chameleon: it disappears, reappears in later builds, then disappears again. If it becomes persistent again, will open another issue. |
|
Or reopen this issue... |
|
It still has been happening quite frequently (probably our most frequent source of failure aside from networking issues), but I think someone recently deleted the test? |
Skipped in #50071, but the root problem is still there |
|
We should try re-enabling that, as the underlying problem is fixed now |
|
Tests restored in #55368 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
E.g. https://buildkite.com/julialang/julia-master/builds/15140#0182e880-cff6-49c2-a275-2164b2b9c41e, but I've seen it in other builds also:
The text was updated successfully, but these errors were encountered: