Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[mono][linux][x64] crash in ep_buffer_manager_write_event #84257

Closed
carlossanlop opened this issue Apr 3, 2023 · 7 comments
Closed

[mono][linux][x64] crash in ep_buffer_manager_write_event #84257

carlossanlop opened this issue Apr 3, 2023 · 7 comments
Labels
arch-x64 area-Diagnostics-mono blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' EventPipe Known Build Error Use this to report build issues in the .NET Helix tab os-linux Linux OS (any supported distro) runtime-mono specific to the Mono runtime

Comments

@carlossanlop
Copy link
Member

carlossanlop commented Apr 3, 2023 •
edited by build-analysis bot
Loading

Build Information

Found in this main PR: #84102
Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=223610
Failing test: System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest
Queue: Libraries Test Run release mono_interpreter linux x64 Debug
Job: https://dev.azure.com/dnceng-public/public/_build/results?buildId=23610&view=logs&j=9310185d-988f-5930-4a09-6918c1bb36fb&t=70410075-93d6-55d0-0186-00493ab8b56c&l=54
Log file: https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-84102-merge-48339c13fe3041a4a0/System.Threading.ThreadPool.Tests/1/console.0ce7c671.log?helixlogtype=result

Error Message

Output example (expand) 
----- start Fri 31 Mar 2023 01:11:41 AM UTC =============== To repro directly: =====================================================
pushd .
/root/helix/work/correlation/dotnet exec --runtimeconfig System.Threading.ThreadPool.Tests.runtimeconfig.json --depsfile System.Threading.ThreadPool.Tests.deps.json xunit.console.dll System.Threading.ThreadPool.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
  Discovering: System.Threading.ThreadPool.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Threading.ThreadPool.Tests (found 44 of 45 test cases)
  Starting:    System.Threading.ThreadPool.Tests (parallel test collections = on, max threads = 2)

=================================================================

=================================================================
	Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
	Native stacktrace:
=================================================================
	0x7fdfb3d208e2 - Unknown
	0x7fdfb3cc5e3e - Unknown
	0x7fdfb3c23091 - Unknown
	0x7fdfb4544730 - Unknown
	0x7fdfb3d45267 - Unknown
	0x7fdfb3d54a84 - Unknown
	0x7fdfb3d545ae - Unknown
	0x7fdfb3d6a2f0 - Unknown
	0x7fdfb3d6a46b - Unknown
	0x7fdfb3d3152f - Unknown
	0x7fdfb3d35f8f - Unknown
	0x7fdfb3d3696d - Unknown
	0x4103b90f - Unknown
	Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 

=================================================================
	External Debugger Dump:
=================================================================
used by your application.
=================================================================

An error has occurred in the native fault reporting. Some diagnostic information will be unavailable.

=================================================================
	Native stacktrace:
=================================================================
	0x7fdfb3d208e2 - Unknown
	0x7fdfb3cc5e3e - Unknown
	0x7fdfb3c23091 - Unknown
	0x7fdfb4544730 - Unknown
	0x7fdfb3e750a6 - Unknown
	0x7fdfb3de924f - Unknown
	0x7fdfb3d9754e - Unknown
	0x7fdfb3d9c630 - Unknown
	0x7fdfb3d947c2 - Unknown
	0x7fdfb3d943f2 - Unknown
	0x7fdfb3daa6a4 - Unknown
	0x7fdfb3e6906d - Unknown
	0x7fdfb3e6b7eb - Unknown
	0x7fdfb3e706be - Unknown
	0x7fdfb3e70449 - Unknown
	0x7fdfb4539fa3 - Unknown
	0x7fdfb414506f - Unknown

=================================================================
	External Debugger Dump:
=================================================================
[New LWP 210]
[New LWP 211]
[New LWP 212]
[New LWP 213]
/tmp/mono-gdb-commands.209:1: Error in sourced command file:
warning: process 209 is already traced by process 216
ptrace: Operation not permitted.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007fdfb45440ca in __waitpid (pid=215, stat_loc=0x7ffcffb1e440, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
30	../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.
  Id   Target Id                                         Frame 
* 1    Thread 0x7fdfb4047740 (LWP 209) "dotnet"          0x00007fdfb45440ca in __waitpid (pid=215, stat_loc=0x7ffcffb1e440, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
  2    Thread 0x7fdfb2fff700 (LWP 210) "SGen worker"     futex_wait_cancelable (private=0, expected=0, futex_word=0x7fdfb3f1f6e8 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  3    Thread 0x7fdfb1408700 (LWP 211) ".NET EventPipe"  0x00007fdfb413a6f9 in __GI___poll (fds=0x7fdfac002710, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
  4    Thread 0x7fdfb1207700 (LWP 212) "Finalizer"       futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7fdfb3f11088 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  5    Thread 0x7fdfb09ed700 (LWP 213) ".NET Long Runni" 0x00007fdfb45440ca in __waitpid (pid=216, stat_loc=0x7fdfb09eadc0, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30

Thread 5 (Thread 0x7fdfb09ed700 (LWP 213)):
#0  0x00007fdfb45440ca in __waitpid (pid=216, stat_loc=0x7fdfb09eadc0, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1  0x00007fdfb3d20a27 in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:843
#2  mono_dump_native_crash_info (signal=<optimized out>, mctx=0x7fdfb09eb930, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:870
#3  0x00007fdfb3cc5e3e in mono_handle_native_crash (signal=0x7fdfb3b6ca11 "SIGSEGV", mctx=0x7fdfb09eb930, info=0x7fdfb09ebbf0) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2979
#4  0x00007fdfb3c23091 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0x7fdfb09ebbf0, context=0x7fdfb09ebac0, debug_fault_addr=0x7fe56c49fbc0) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3758
#5  <signal handler called>
#6  0x00007fdfb3e750a6 in mono_w32handle_ops_typename (type=<optimized out>) at /__w/1/s/src/mono/mono/metadata/w32handle.c:452
#7  mono_w32handle_ref_core (handle_data=0x55aab720f680) at /__w/1/s/src/mono/mono/metadata/w32handle.c:334
#8  mono_w32handle_lookup_and_ref (handle=0x55aab720f680, handle_data=0x7fdfb09ec7f0) at /__w/1/s/src/mono/mono/metadata/w32handle.c:309
#9  0x00007fdfb3de924f in mono_w32event_set (handle=0x55aab720f680) at /__w/1/s/src/mono/mono/metadata/w32event-unix.c:170
#10 0x00007fdfb3d9754e in ep_rt_wait_event_set (wait_event=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/eventpipe/ep-rt-mono.h:707
#11 ep_buffer_manager_write_event (buffer_manager=<optimized out>, thread=<optimized out>, session=<optimized out>, ep_event=<optimized out>, payload=<optimized out>, activity_id=<optimized out>, related_activity_id=<optimized out>, event_thread=<optimized out>, stack=<optimized out>) at /__w/1/s/src/native/eventpipe/ep-buffer-manager.c:1031
#12 0x00007fdfb3d9c630 in ep_session_write_event (session=0x55aab7131090, thread=0x7fdfb09ec7f0, ep_event=0xb70b1e90, payload=0x7fdfb09ecd08, activity_id=0x7fdfa0002670 "", related_activity_id=0x0, event_thread=<optimized out>, stack=<optimized out>) at /__w/1/s/src/native/eventpipe/ep-session.c:488
#13 0x00007fdfb3d947c2 in write_event_2 (thread=<optimized out>, ep_event=<optimized out>, payload=<optimized out>, activity_id=<optimized out>, related_activity_id=<optimized out>, event_thread=0x0, stack=<optimized out>) at /__w/1/s/src/native/eventpipe/ep.c:773
#14 0x00007fdfb3d943f2 in write_event (ep_event=0x55aab6b8d7f0, payload=0x7fdfb09ecd08, activity_id=0x7fdfa0002670 "", related_activity_id=0x0) at /__w/1/s/src/native/eventpipe/ep.c:707
#15 ep_write_event (ep_event=0x55aab6b8d7f0, data=<optimized out>, data_len=<optimized out>, activity_id=<optimized out>, related_activity_id=0x0) at /__w/1/s/src/native/eventpipe/ep.c:1500
#16 0x00007fdfb3daa6a4 in EventPipeWriteEventThreadTerminated (ManagedThreadID=<optimized out>, AppDomainID=<optimized out>, ClrInstanceID=9, ActivityId=0x0, RelatedActivityId=0x0) at eventpipe/dotnetruntime.c:1305
#17 0x00007fdfb3e6906d in mono_profiler_raise_thread_stopped (tid=140598717634304) at /__w/1/s/src/native/public/./mono/metadata/profiler-events.h:104
#18 0x00007fdfb3e6b7eb in mono_thread_detach_internal (thread=0x7fdfb30731d0) at /__w/1/s/src/mono/mono/metadata/threads.c:1026
#19 0x00007fdfb3e706be in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1245
#20 0x00007fdfb3e70449 in start_wrapper (data=0x55aab718b660) at /__w/1/s/src/mono/mono/metadata/threads.c:1264
#21 0x00007fdfb4539fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#22 0x00007fdfb414506f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7fdfb1207700 (LWP 212)):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7fdfb3f11088 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x7fdfb3f11088 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:111
#2  0x00007fdfb4542988 in __new_sem_wait_slow (sem=0x7fdfb3f11088 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007fdfb3e981d6 in mono_os_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/metadata/../utils/mono-os-semaphore.h:204
#4  mono_coop_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/metadata/../../mono/utils/mono-coop-semaphore.h:41
#5  finalizer_thread (unused=<optimized out>) at /__w/1/s/src/mono/mono/metadata/gc.c:891
#6  0x00007fdfb3e705ba in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1202
#7  0x00007fdfb3e70449 in start_wrapper (data=0x55aab6bc6d30) at /__w/1/s/src/mono/mono/metadata/threads.c:1264
#8  0x00007fdfb4539fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9  0x00007fdfb414506f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7fdfb1408700 (LWP 211)):
#0  0x00007fdfb413a6f9 in __GI___poll (fds=0x7fdfac002710, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007fdfb3db8bba in ipc_poll_fds (fds=<optimized out>, nfds=1, timeout=4294967295) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:470
#2  ds_ipc_poll (poll_handles_data=0x7fdfac002500, poll_handles_data_len=1, timeout_ms=4294967295, callback=0x7fdfb3db8020 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:1098
#3  0x00007fdfb3db60c5 in ds_ipc_stream_factory_get_next_available_stream (callback=<optimized out>) at /__w/1/s/src/native/eventpipe/ds-ipc.c:385
#4  0x00007fdfb3db46ad in server_thread (data=<optimized out>) at /__w/1/s/src/native/eventpipe/ds-server.c:129
#5  0x00007fdfb3db8001 in ep_rt_thread_mono_start_func (data=0x55aab6b9d300) at /__w/1/s/src/mono/mono/mini/../eventpipe/ep-rt-mono.h:880
#6  0x00007fdfb4539fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007fdfb414506f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7fdfb2fff700 (LWP 210)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7fdfb3f1f6e8 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fdfb3f1f698 <lock>, cond=0x7fdfb3f1f6c0 <work_cond>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fdfb3f1f6c0 <work_cond>, mutex=0x7fdfb3f1f698 <lock>) at pthread_cond_wait.c:655
#3  0x00007fdfb3ef85c3 in mono_os_cond_wait (cond=<optimized out>, mutex=<optimized out>) at /__w/1/s/src/mono/mono/sgen/../../mono/utils/mono-os-mutex.h:219
#4  get_work (worker_index=<optimized out>, work_context=<optimized out>, do_idle=<optimized out>, job=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:167
#5  thread_func (data=0x0) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:198
#6  0x00007fdfb4539fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007fdfb414506f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7fdfb4047740 (LWP 209)):
#0  0x00007fdfb45440ca in __waitpid (pid=215, stat_loc=0x7ffcffb1e440, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1  0x00007fdfb3d20a27 in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:843
#2  mono_dump_native_crash_info (signal=<optimized out>, mctx=0x7ffcffb1efb0, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:870
#3  0x00007fdfb3cc5e3e in mono_handle_native_crash (signal=0x7fdfb3b6ca11 "SIGSEGV", mctx=0x7ffcffb1efb0, info=0x7ffcffb1f270) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2979
#4  0x00007fdfb3c23091 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0x7ffcffb1f270, context=0x7ffcffb1f140, debug_fault_addr=0x55abb70b1ec0) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3758
#5  <signal handler called>
#6  generate_code (td=<optimized out>, method=<optimized out>, header=<optimized out>, generic_context=0x0, error=<optimized out>) at /__w/1/s/src/mono/mono/mini/interp/transform.c:4849
#7  0x00007fdfb3d54a84 in generate (method=<optimized out>, header=0x55aab7172c80, rtm=0x7ffcffb202f8, generic_context=0x0, error=0x7ffcffb20410) at /__w/1/s/src/mono/mono/mini/interp/transform.c:10853
#8  0x00007fdfb3d545ae in mono_interp_transform_method (imethod=0x7ffcffb202f8, context=<optimized out>, error=0x7ffcffb20410) at /__w/1/s/src/mono/mono/mini/interp/transform.c:11129
#9  0x00007fdfb3d6a2f0 in tier_up_method (imethod=0x55aab6cac9a8, context=0x55aab6b655b0) at /__w/1/s/src/mono/mono/mini/interp/tiering.c:81
#10 0x00007fdfb3d6a46b in mono_interp_tier_up_frame_patchpoint (frame=0x7ffcffb205a0, context=0x55aab6b655b0, bb_index=51) at /__w/1/s/src/mono/mono/mini/interp/tiering.c:192
#11 0x00007fdfb3d3152f in mono_interp_exec_method (frame=<optimized out>, context=0x55aab6b655b0, clause_args=<optimized out>) at /__w/1/s/src/mono/mono/mini/interp/interp.c:7490
#12 0x00007fdfb3d35f8f in interp_entry (data=<optimized out>) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2244
#13 0x00007fdfb3d3696d in interp_entry_static_7 (arg1=<optimized out>, arg2=0xfffffffffffffffe, arg3=0x1, arg4=0x0, arg5=0x55abb70b1e90, arg6=0x55aab7210920, arg7=0x7ffcffb209d0, rmethod=0x55aab6c52368) at /__w/1/s/src/mono/mono/mini/interp/interp.c:3022
#14 0x000000004103b90f in ?? ()
#15 0x00007ffcffb209d0 in ?? ()
#16 0x000055aab6c52368 in ?? ()
#17 0x00000000ffffff00 in ?? ()
#18 0x0000000000000000 in ?? ()
[Inferior 1 (process 209) detached]

=================================================================
	Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x7fdfb3d45267):0x7fdfb3d45257  83 c5 01 83 c3 ff 75 e1 4c 8b 84 24 08 01 00 00  ......u.L..$....
0x7fdfb3d45267  41 f6 40 30 02 0f 85 9e 00 00 00 48 8b 44 24 50  A.@0.......H.D$P
0x7fdfb3d45277  48 8b 00 f6 40 7c 04 0f 85 8c 00 00 00 4d 89 fd  H...@|.......M..
0x7fdfb3d45287  48 8b 6c 24 28 83 bd 38 01 00 00 02 4c 8b bc 24  H.l$(..8....L..$

=================================================================
	Managed Stacktrace:
=================================================================
	  at <unknown> <0xffffffff>
	  at System.Diagnostics.Tracing.EventPipeInternal:Disable <0x00018>
	  at System.Diagnostics.Tracing.EventPipeEventDispatcher:CommitDispatchConfiguration <0x0002a>
	  at System.Diagnostics.Tracing.EventPipeEventDispatcher:RemoveEventListener <0x00068>
	  at System.Diagnostics.Tracing.EventListener:RemoveReferencesToListenerInEventSources <0x00186>
	  at System.Diagnostics.Tracing.EventListener:Dispose <0x00088>
	  at <>c:<ThreadPoolMinMaxThreadsEventTest>b__42_0 <0x0045e>
	  at System.Object:runtime_invoke_direct_void__this__ <0x00098>
	  at <unknown> <0x00000>
	  at <unknown> <0xffffffff>
	  at System.Reflection.RuntimeMethodInfo:InternalInvoke <0x00090>
	  at System.Reflection.MethodInvoker:InterpretedInvoke <0x0003c>
	  at System.Reflection.MethodInvoker:Invoke <0x00108>
	  at System.Reflection.MethodInvoker:InlinedInvoke <0x0007e>
	  at System.Reflection.RuntimeMethodInfo:Invoke <0x000ec>
	  at System.Reflection.MethodBase:Invoke <0x0002e>
	  at Microsoft.DotNet.RemoteExecutor.Program:Main <0x0023c>
	  at <Module>:runtime_invoke_direct_int_string[] <0x000a0>
	  at <unknown> <0x00000>
=================================================================
    System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest [FAIL]
      Exit code was 134 but it should have been 42
      Expected: True
      Actual:   False
      Stack Trace:
        /_/src/Microsoft.DotNet.RemoteExecutor/src/RemoteInvokeHandle.cs(239,0): at Microsoft.DotNet.RemoteExecutor.RemoteInvokeHandle.Dispose(Boolean disposing)
        /_/src/Microsoft.DotNet.RemoteExecutor/src/RemoteInvokeHandle.cs(57,0): at Microsoft.DotNet.RemoteExecutor.RemoteInvokeHandle.Dispose()
        /_/src/libraries/System.Threading.ThreadPool/tests/ThreadPoolTests.cs(1114,0): at System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest()
        /_/src/mono/System.Private.CoreLib/src/System/Reflection/MethodInvoker.Mono.cs(30,0): at System.Reflection.MethodInvoker.InterpretedInvoke(Object obj, IntPtr* args)
        /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodInvoker.cs(59,0): at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
    System.Threading.ThreadPools.Tests.ThreadPoolTests.MetricsTest [SKIP]
      Condition(s) not met: "IsMetricsTestSupported"
  Finished:    System.Threading.ThreadPool.Tests
=== TEST EXECUTION SUMMARY ===
   System.Threading.ThreadPool.Tests  Total: 66, Errors: 0, Failed: 1, Skipped: 1, Time: 17.315s
/root/helix/work/workitem/e
----- end Fri 31 Mar 2023 01:11:59 AM UTC ----- exit code 1 ----------------------------------------------------------

{
  "ErrorMessage": "mono_w32handle_ops_typename",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Report

Build Definition Test Pull Request
269142 dotnet/runtime PayloadGroup0.WorkItemExecution #86050
268718 dotnet/runtime PayloadGroup0.WorkItemExecution
268078 dotnet/runtime PayloadGroup0.WorkItemExecution
267827 dotnet/runtime PayloadGroup0.WorkItemExecution
267715 dotnet/runtime PayloadGroup0.WorkItemExecution #85908
267823 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest
267487 dotnet/runtime System.Reflection.Metadata.ApplyUpdateTest.CustomAttributeDelete #85796
267007 dotnet/runtime PayloadGroup0.WorkItemExecution
266742 dotnet/runtime PayloadGroup0.WorkItemExecution
266495 dotnet/runtime PayloadGroup0.WorkItemExecution
265629 dotnet/runtime PayloadGroup0.WorkItemExecution #85908
266409 dotnet/runtime PayloadGroup0.WorkItemExecution
265549 dotnet/runtime PayloadGroup0.WorkItemExecution
265427 dotnet/runtime PayloadGroup0.WorkItemExecution #85810
265291 dotnet/runtime PayloadGroup0.WorkItemExecution
265057 dotnet/runtime PayloadGroup0.WorkItemExecution
264846 dotnet/runtime PayloadGroup0.WorkItemExecution
264752 dotnet/runtime PayloadGroup0.WorkItemExecution #85852
264740 dotnet/runtime PayloadGroup0.WorkItemExecution #85847
264749 dotnet/runtime PayloadGroup0.WorkItemExecution #85849
264750 dotnet/runtime PayloadGroup0.WorkItemExecution #85850
264751 dotnet/runtime PayloadGroup0.WorkItemExecution #85851
264500 dotnet/runtime PayloadGroup0.WorkItemExecution
264235 dotnet/runtime PayloadGroup0.WorkItemExecution #85692
264236 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #85692
264051 dotnet/runtime PayloadGroup0.WorkItemExecution
263902 dotnet/runtime PayloadGroup0.WorkItemExecution #85185
263342 dotnet/runtime PayloadGroup0.WorkItemExecution
262840 dotnet/runtime PayloadGroup0.WorkItemExecution
262880 dotnet/runtime System.Reflection.Metadata.ApplyUpdateTest.ClassWithCustomAttributes #85796
262687 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest
261788 dotnet/runtime PayloadGroup0.WorkItemExecution #84566
261990 dotnet/runtime PayloadGroup0.WorkItemExecution
261437 dotnet/runtime PayloadGroup0.WorkItemExecution
261527 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #84132
261188 dotnet/runtime PayloadGroup0.WorkItemExecution #85692
260534 dotnet/runtime PayloadGroup0.WorkItemExecution
259956 dotnet/runtime tracing/eventpipe/bigevent/bigevent/bigevent.sh
258928 dotnet/runtime PayloadGroup0.WorkItemExecution #84566
258971 dotnet/runtime PayloadGroup0.WorkItemExecution
258590 dotnet/runtime PayloadGroup0.WorkItemExecution
258289 dotnet/runtime PayloadGroup0.WorkItemExecution #85601
257874 dotnet/runtime PayloadGroup0.WorkItemExecution
257815 dotnet/runtime PayloadGroup0.WorkItemExecution #85504
257694 dotnet/runtime PayloadGroup0.WorkItemExecution
257380 dotnet/runtime PayloadGroup0.WorkItemExecution
257197 dotnet/runtime PayloadGroup0.WorkItemExecution
257000 dotnet/runtime PayloadGroup0.WorkItemExecution
256625 dotnet/runtime tracing/eventpipe/providervalidation/providervalidation/providervalidation.sh
256353 dotnet/runtime PayloadGroup0.WorkItemExecution
255903 dotnet/runtime PayloadGroup0.WorkItemExecution #85458
255744 dotnet/runtime PayloadGroup0.WorkItemExecution
255582 dotnet/runtime PayloadGroup0.WorkItemExecution
255331 dotnet/runtime PayloadGroup0.WorkItemExecution
255110 dotnet/runtime PayloadGroup0.WorkItemExecution
255049 dotnet/runtime PayloadGroup0.WorkItemExecution #85404
254950 dotnet/runtime PayloadGroup0.WorkItemExecution
254685 dotnet/runtime PayloadGroup0.WorkItemExecution
254313 dotnet/runtime PayloadGroup0.WorkItemExecution
253841 dotnet/runtime PayloadGroup0.WorkItemExecution
253113 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #85308
252815 dotnet/runtime PayloadGroup0.WorkItemExecution #85185
252256 dotnet/runtime PayloadGroup0.WorkItemExecution
252361 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #85310
251200 dotnet/runtime tracing/eventpipe/buffersize/buffersize/buffersize.sh
251038 dotnet/runtime PayloadGroup0.WorkItemExecution #85235

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 0 66
@carlossanlop carlossanlop added os-linux Linux OS (any supported distro) arch-x64 blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' area-VM-meta-mono runtime-mono specific to the Mono runtime Known Build Error Use this to report build issues in the .NET Helix tab labels Apr 3, 2023
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Apr 3, 2023
@MihaZupan MihaZupan mentioned this issue Apr 13, 2023
Merged
@steveisok
Copy link
Member

/cc @lambdageek @BrzVlad

@lambdageek
Copy link
Member

"waitpid.c: No such file or directory" is way too broad of a filter. that is marking every single failure that results in a native crash on Linux as a known issue

@lambdageek
Copy link
Member

I think the original issue (judging by the sample crash report) is due to mono_w32handle_ops_typename crashing.

@lambdageek
Copy link
Member

lambdageek commented May 10, 2023 •
edited
Loading

@lateralusX some kind of eventpipe failure. (see "Output example (expand)" dropdown in the issue description)

Thread 5 (Thread 0x7fdfb09ed700 (LWP 213)):
#0  0x00007fdfb45440ca in __waitpid (pid=216, stat_loc=0x7fdfb09eadc0, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1  0x00007fdfb3d20a27 in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:843
#2  mono_dump_native_crash_info (signal=<optimized out>, mctx=0x7fdfb09eb930, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:870
#3  0x00007fdfb3cc5e3e in mono_handle_native_crash (signal=0x7fdfb3b6ca11 "SIGSEGV", mctx=0x7fdfb09eb930, info=0x7fdfb09ebbf0) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2979
#4  0x00007fdfb3c23091 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0x7fdfb09ebbf0, context=0x7fdfb09ebac0, debug_fault_addr=0x7fe56c49fbc0) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3758
#5  <signal handler called>
#6  0x00007fdfb3e750a6 in mono_w32handle_ops_typename (type=<optimized out>) at /__w/1/s/src/mono/mono/metadata/w32handle.c:452
#7  mono_w32handle_ref_core (handle_data=0x55aab720f680) at /__w/1/s/src/mono/mono/metadata/w32handle.c:334
#8  mono_w32handle_lookup_and_ref (handle=0x55aab720f680, handle_data=0x7fdfb09ec7f0) at /__w/1/s/src/mono/mono/metadata/w32handle.c:309
#9  0x00007fdfb3de924f in mono_w32event_set (handle=0x55aab720f680) at /__w/1/s/src/mono/mono/metadata/w32event-unix.c:170
#10 0x00007fdfb3d9754e in ep_rt_wait_event_set (wait_event=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/eventpipe/ep-rt-mono.h:707
#11 ep_buffer_manager_write_event (buffer_manager=<optimized out>, thread=<optimized out>, session=<optimized out>, ep_event=<optimized out>, payload=<optimized out>, activity_id=<optimized out>, related_activity_id=<optimized out>, event_thread=<optimized out>, stack=<optimized out>) at /__w/1/s/src/native/eventpipe/ep-buffer-manager.c:1031
#12 0x00007fdfb3d9c630 in ep_session_write_event (session=0x55aab7131090, thread=0x7fdfb09ec7f0, ep_event=0xb70b1e90, payload=0x7fdfb09ecd08, activity_id=0x7fdfa0002670 "", related_activity_id=0x0, event_thread=<optimized out>, stack=<optimized out>) at /__w/1/s/src/native/eventpipe/ep-session.c:488
#13 0x00007fdfb3d947c2 in write_event_2 (thread=<optimized out>, ep_event=<optimized out>, payload=<optimized out>, activity_id=<optimized out>, related_activity_id=<optimized out>, event_thread=0x0, stack=<optimized out>) at /__w/1/s/src/native/eventpipe/ep.c:773
#14 0x00007fdfb3d943f2 in write_event (ep_event=0x55aab6b8d7f0, payload=0x7fdfb09ecd08, activity_id=0x7fdfa0002670 "", related_activity_id=0x0) at /__w/1/s/src/native/eventpipe/ep.c:707
#15 ep_write_event (ep_event=0x55aab6b8d7f0, data=<optimized out>, data_len=<optimized out>, activity_id=<optimized out>, related_activity_id=0x0) at /__w/1/s/src/native/eventpipe/ep.c:1500
#16 0x00007fdfb3daa6a4 in EventPipeWriteEventThreadTerminated (ManagedThreadID=<optimized out>, AppDomainID=<optimized out>, ClrInstanceID=9, ActivityId=0x0, RelatedActivityId=0x0) at eventpipe/dotnetruntime.c:1305
#17 0x00007fdfb3e6906d in mono_profiler_raise_thread_stopped (tid=140598717634304) at /__w/1/s/src/native/public/./mono/metadata/profiler-events.h:104
#18 0x00007fdfb3e6b7eb in mono_thread_detach_internal (thread=0x7fdfb30731d0) at /__w/1/s/src/mono/mono/metadata/threads.c:1026
#19 0x00007fdfb3e706be in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1245
#20 0x00007fdfb3e70449 in start_wrapper (data=0x55aab718b660) at /__w/1/s/src/mono/mono/metadata/threads.c:1264
#21 0x00007fdfb4539fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#22 0x00007fdfb414506f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

@lambdageek lambdageek changed the title [mono][linux][x64] Native crash: waitpd.c no such file or directory [mono][linux][x64] crash in ep_buffer_manager_write_event May 10, 2023
@lateralusX
Copy link
Member

lateralusX commented May 11, 2023 •
edited
Loading

Sound like it could be related to a couple of other similar scenarios, #85213, #83575, those manifests on CoreCLR under debug build (triggers an assert) but it seems like there is a race during disable of a session and having a thread writing into the session at the same time as closing it down.

@lateralusX lateralusX mentioned this issue May 15, 2023
Merged
@lateralusX
Copy link
Member

lateralusX commented May 15, 2023 •
edited
Loading

There is a pending fix in #86180 that most likely will fix this issue as well. Currently a session can close down while there are pending threads writing events into it. Looks like this issue triggers the same scenario as fixed by that PR.

@lateralusX
Copy link
Member

0 in the 7-day count after #86180 got merged, close this as fixed by #86180.

@ghost ghost removed the untriaged New issue has not been triaged by the area owner label May 23, 2023
@ghost ghost locked as resolved and limited conversation to collaborators Jun 22, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
arch-x64 area-Diagnostics-mono blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' EventPipe Known Build Error Use this to report build issues in the .NET Helix tab os-linux Linux OS (any supported distro) runtime-mono specific to the Mono runtime
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@steveisok @lambdageek @carlossanlop @lateralusX