Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mono AOT Arm64 failing #2625

Closed
DrewScoggins opened this issue Oct 3, 2022 · 5 comments · Fixed by dotnet/BenchmarkDotNet#2134 or #2639
Closed

Mono AOT Arm64 failing #2625

DrewScoggins opened this issue Oct 3, 2022 · 5 comments · Fixed by dotnet/BenchmarkDotNet#2134 or #2639
Assignees
@naricc

Comments

@DrewScoggins
Copy link
Member

Below is the output of running the Arm64 Mono AOT runs on the new Ampere hardware. These machines are running Ubuntu 20.04.

[2022/10/03 05:42:06][INFO] // **************************
[2022/10/03 05:42:06][INFO] // Benchmark: LinqBenchmarks.Count00ForX: Job-EUPFDP(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=MonoAOTLLVM, Toolchain=MonoAOTLLVM, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1)
[2022/10/03 05:42:06][INFO] // *** Execute ***
[2022/10/03 05:42:06][INFO] // Launch: 1 / 1
[2022/10/03 05:42:06][INFO] // Execute: /home/helixbot/work/A5400950/w/99B20887/e/performance/artifacts/bin/MicroBenchmarks/Release/net7.0/0e3e1f9c-24a4-4379-9da8-6afaf19d954f/bin/net7.0/linux-arm64/publish/0e3e1f9c-24a4-4379-9da8-6afaf19d954f --benchmarkName LinqBenchmarks.Count00ForX --job "PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=MonoAOTLLVM, Toolchain=MonoAOTLLVM, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1" --benchmarkId 1 in /home/helixbot/work/A5400950/w/99B20887/e/performance/artifacts/bin/MicroBenchmarks/Release/net7.0/0e3e1f9c-24a4-4379-9da8-6afaf19d954f/bin/net7.0/linux-arm64/publish
[2022/10/03 05:42:06][INFO] Failed to set up high priority. Make sure you have the right permissions. Message: Permission denied
[2022/10/03 05:42:06][INFO] LLVM JIT not supported on this platform.
[2022/10/03 05:42:06][INFO] 
[2022/10/03 05:42:06][INFO] =================================================================
[2022/10/03 05:42:06][INFO] 	Native Crash Reporting
[2022/10/03 05:42:06][INFO] =================================================================
[2022/10/03 05:42:06][INFO] Got a SIGABRT while executing native code. This usually indicates
[2022/10/03 05:42:06][INFO] a fatal error in the mono runtime or one of the native libraries
[2022/10/03 05:42:06][INFO] used by your application.
[2022/10/03 05:42:06][INFO] =================================================================
[2022/10/03 05:42:06][INFO] 
[2022/10/03 05:42:06][INFO] =================================================================
[2022/10/03 05:42:06][INFO] 	Native stacktrace:
[2022/10/03 05:42:06][INFO] =================================================================
[2022/10/03 05:42:06][INFO] 	0xffff921dc9a8 - Unknown
[2022/10/03 05:42:06][INFO] 
[2022/10/03 05:42:06][INFO] =================================================================
[2022/10/03 05:42:06][INFO] 	External Debugger Dump:
[2022/10/03 05:42:06][INFO] =================================================================
[2022/10/03 05:42:06][INFO] [New LWP 86527]
[2022/10/03 05:42:06][INFO] [New LWP 86528]
[2022/10/03 05:42:06][INFO] [Thread debugging using libthread_db enabled]
[2022/10/03 05:42:06][INFO] Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
[2022/10/03 05:42:07][INFO] 0x0000ffff92bb1c0c in __GI___wait4 (pid=<optimized out>, stat_loc=0xffffc313f820, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27
[2022/10/03 05:42:07][INFO] 27	../sysdeps/unix/sysv/linux/wait4.c: No such file or directory.
[2022/10/03 05:42:07][INFO]   Id   Target Id                                           Frame
[2022/10/03 05:42:07][INFO] * 1    Thread 0xffff92fb5010 (LWP 86526) "0e3e1f9c-24a4-4" 0x0000ffff92bb1c0c in __GI___wait4 (pid=<optimized out>, stat_loc=0xffffc313f820, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27
[2022/10/03 05:42:07][INFO]   2    Thread 0xffff90fff1b0 (LWP 86527) "SGen worker"     futex_wait_cancelable (private=0, expected=0, futex_word=0xffff92a18328 <work_cond+40>) at ../sysdeps/nptl/futex-internal.h:183
[2022/10/03 05:42:07][INFO]   3    Thread 0xffff8ec561b0 (LWP 86528) ".NET EventPipe"  0x0000ffff92bd5f08 in __GI___poll (fds=0xffff88003b20, nfds=1, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
[2022/10/03 05:42:07][INFO] 
[2022/10/03 05:42:07][INFO] Thread 3 (Thread 0xffff8ec561b0 (LWP 86528)):
[2022/10/03 05:42:07][INFO] #0  0x0000ffff92bd5f08 in __GI___poll (fds=0xffff88003b20, nfds=1, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
[2022/10/03 05:42:07][INFO] #1  0x0000ffff922875c4 in ipc_poll_fds (fds=<optimized out>, nfds=1, timeout=4294967295) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:470
[2022/10/03 05:42:07][INFO] #2  ds_ipc_poll (poll_handles_data=0xffff88003310, poll_handles_data_len=1, timeout_ms=4294967295, callback=0xffff92286974 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:1096
[2022/10/03 05:42:07][INFO] #3  0x0000ffff92284b70 in ds_ipc_stream_factory_get_next_available_stream (callback=0xffff92286974 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc.c:395
[2022/10/03 05:42:07][INFO] #4  0x0000ffff922833e8 in server_thread (data=<optimized out>) at /__w/1/s/src/native/eventpipe/ds-server.c:129
[2022/10/03 05:42:07][INFO] #5  0x0000ffff92286954 in ep_rt_thread_mono_start_func (data=0xaaab07d19030) at /__w/1/s/src/mono/mono/mini/../eventpipe/ep-rt-mono.h:1332
[2022/10/03 05:42:07][INFO] #6  0x0000ffff92f50624 in start_thread (arg=0xffff92286924 <ep_rt_thread_mono_start_func>) at pthread_create.c:477
[2022/10/03 05:42:07][INFO] #7  0x0000ffff92bdf49c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
[2022/10/03 05:42:07][INFO] 
[2022/10/03 05:42:07][INFO] Thread 2 (Thread 0xffff90fff1b0 (LWP 86527)):
[2022/10/03 05:42:07][INFO] #0  futex_wait_cancelable (private=0, expected=0, futex_word=0xffff92a18328 <work_cond+40>) at ../sysdeps/nptl/futex-internal.h:183
[2022/10/03 05:42:07][INFO] #1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xffff92a182d0 <lock>, cond=0xffff92a18300 <work_cond>) at pthread_cond_wait.c:508
[2022/10/03 05:42:07][INFO] #2  __pthread_cond_wait (cond=0xffff92a18300 <work_cond>, mutex=0xffff92a182d0 <lock>) at pthread_cond_wait.c:647
[2022/10/03 05:42:07][INFO] #3  0x0000ffff920e5f38 in mono_os_cond_wait (cond=0xffff92a18328 <work_cond+40>, mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-os-mutex.h:219
[2022/10/03 05:42:07][INFO] #4  get_work (worker_index=<optimized out>, work_context=<optimized out>, do_idle=<optimized out>, job=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:167
[2022/10/03 05:42:07][INFO] #5  thread_func (data=0x0) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:198
[2022/10/03 05:42:07][INFO] #6  0x0000ffff92f50624 in start_thread (arg=0xffff920e5e38 <thread_func>) at pthread_create.c:477
[2022/10/03 05:42:07][INFO] #7  0x0000ffff92bdf49c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
[2022/10/03 05:42:07][INFO] 
[2022/10/03 05:42:07][INFO] Thread 1 (Thread 0xffff92fb5010 (LWP 86526)):
[2022/10/03 05:42:07][INFO] #0  0x0000ffff92bb1c0c in __GI___wait4 (pid=<optimized out>, stat_loc=0xffffc313f820, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27
[2022/10/03 05:42:07][INFO] #1  0x0000ffff921dcaa8 in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:843
[2022/10/03 05:42:07][INFO] #2  mono_dump_native_crash_info (signal=<optimized out>, mctx=0xffffc3140250, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:870
[2022/10/03 05:42:07][INFO] #3  0x0000ffff92196678 in mono_handle_native_crash (signal=0xffff91ec8e46 "SIGABRT", mctx=0xffffc3140250, info=0xffffc3140590) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:3005
[2022/10/03 05:42:07][INFO] #4  0x0000ffff921dc1b4 in sigabrt_signal_handler (_dummy=6, _info=0xffffc3140590, context=0xffffc3140610) at /__w/1/s/src/mono/mono/mini/mini-posix.c:225
[2022/10/03 05:42:07][INFO] #5  <signal handler called>
[2022/10/03 05:42:07][INFO] #6  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
[2022/10/03 05:42:07][INFO] #7  0x0000ffff92b2eaac in __GI_abort () at abort.c:79
[2022/10/03 05:42:07][INFO] #8  0x0000ffff9223a76c in monoeg_assert_abort () at /__w/1/s/src/mono/mono/eglib/goutput.c:57
[2022/10/03 05:42:07][INFO] #9  0x0000ffff92091d54 in mono_log_write_logfile (log_domain=<optimized out>, level=G_LOG_LEVEL_ERROR, hdr=<optimized out>, message=0xaaab07d149a0 "LLVM JIT not supported on this platform.") at /__w/1/s/src/mono/mono/utils/mono-log-common.c:146
[2022/10/03 05:42:07][INFO] #10 0x0000ffff9223ab1c in monoeg_g_logstr (log_domain=0x0, log_level=G_LOG_LEVEL_ERROR, msg=0x0) at /__w/1/s/src/mono/mono/eglib/goutput.c:151
[2022/10/03 05:42:07][INFO] #11 monoeg_g_logv_nofree (log_domain=0x0, log_level=G_LOG_LEVEL_ERROR, format=<optimized out>, args=...) at /__w/1/s/src/mono/mono/eglib/goutput.c:166
[2022/10/03 05:42:07][INFO] #12 0x0000ffff9223ab90 in monoeg_g_logv (log_domain=0x0, log_level=-1022093288, format=0x0, args=...) at /__w/1/s/src/mono/mono/eglib/goutput.c:173
[2022/10/03 05:42:07][INFO] #13 monoeg_g_log (log_domain=0x0, log_level=-1022093288, format=0x0) at /__w/1/s/src/mono/mono/eglib/goutput.c:182
[2022/10/03 05:42:07][INFO] #14 0x0000ffff92235918 in mono_llvm_create_ee (ee=<optimized out>) at /__w/1/s/src/mono/mono/mini/llvm-jit.cpp:476
[2022/10/03 05:42:07][INFO] #15 0x0000ffff9221637c in init_jit_module () at /__w/1/s/src/mono/mono/mini/mini-llvm.c:14025
[2022/10/03 05:42:07][INFO] #16 mono_llvm_emit_method (cfg=0xaaab07d1ce00) at /__w/1/s/src/mono/mono/mini/mini-llvm.c:11714
[2022/10/03 05:42:07][INFO] #17 0x0000ffff920f73d0 in mini_method_compile (method=0xaaab07d19e00, opts=374417919, flags=JIT_FLAG_RUN_CCTORS, parts=0, aot_method_index=-1) at /__w/1/s/src/mono/mono/mini/mini.c:3881
[2022/10/03 05:42:07][INFO] #18 0x0000ffff920f892c in mono_jit_compile_method_inner (method=0xaaab07d19e00, opt=-1022093288, error=0xffffc3142cf0) at /__w/1/s/src/mono/mono/mini/mini.c:4107
[2022/10/03 05:42:07][INFO] #19 0x0000ffff921029c4 in mono_jit_compile_method_with_opt (method=0xaaab07d19e00, opt=<optimized out>, jit_only=<optimized out>, error=0xffffc3142cf0) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2707
[2022/10/03 05:42:07][INFO] #20 jit_compile_method_with_opt_cb (arg=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2762
[2022/10/03 05:42:07][INFO] #21 0x0000ffff92235ae0 in mono_llvm_cpp_catch_exception (cb=0x87, arg=0x0, out_thrown=0xffffc3142778) at /__w/1/s/src/mono/mono/mini/llvm-runtime.cpp:36
[2022/10/03 05:42:07][INFO] #22 0x0000ffff92100af8 in jit_compile_method_with_opt (params=0xffffc3142788) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2776
[2022/10/03 05:42:07][INFO] #23 mono_jit_compile_method (method=<optimized out>, error=0xffffc3142cf0) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2797
[2022/10/03 05:42:07][INFO] #24 create_runtime_invoke_info (method=0xaaab07d0e768, compiled_method=<optimized out>, callee_gsharedvt=<optimized out>, use_interp=<optimized out>, error=0xffffc3142cf0) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3207
[2022/10/03 05:42:07][INFO] #25 mono_jit_runtime_invoke (method=0xaaab07d0e768, obj=<optimized out>, params=0xffffc3142c00, exc=0x0, error=0xffffc3142cf0) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3425
[2022/10/03 05:42:07][INFO] #26 0x0000ffff9202b018 in do_runtime_invoke (method=0xaaab07d0e768, obj=0xffff91000048, params=0xffffc3142c00, exc=0x0, error=0xffffc3142cf0) at /__w/1/s/src/mono/mono/metadata/object.c:2415
[2022/10/03 05:42:07][INFO] #27 mono_runtime_invoke_checked (method=0xaaab07d0e768, obj=0xffff91000048, params=0xffffc3142c00, error=0xffffc3142cf0) at /__w/1/s/src/mono/mono/metadata/object.c:2583
[2022/10/03 05:42:07][INFO] #28 0x0000ffff91fe46b4 in create_exception_two_strings (klass=0xaaab07d0e320, a1=..., a2=..., error=0xffffc3142cf0) at /__w/1/s/src/mono/mono/metadata/exception.c:176
[2022/10/03 05:42:07][INFO] #29 0x0000ffff91fe4494 in mono_exception_from_name_two_strings_checked (image=0xaaab07c406f0, name_space=0xffff91e798e6 "System", name=<optimized out>, a1=..., a2=..., error=0xffffc3142cf0) at /__w/1/s/src/mono/mono/metadata/exception.c:236
[2022/10/03 05:42:07][INFO] #30 0x0000ffff91fc18a8 in create_domain_objects (domain=0xaaab07c3bec0) at /__w/1/s/src/mono/mono/metadata/appdomain.c:192
[2022/10/03 05:42:07][INFO] #31 mono_runtime_init_checked (domain=0xaaab07c3bec0, start_cb=<optimized out>, attach_cb=<optimized out>, error=0xffffc3142e88) at /__w/1/s/src/mono/mono/metadata/appdomain.c:288
[2022/10/03 05:42:07][INFO] #32 0x0000ffff920ffad0 in mini_init (filename=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:4722
[2022/10/03 05:42:07][INFO] #33 0x0000ffff92159e0c in mono_main (argc=<optimized out>, argv=0x87) at /__w/1/s/src/mono/mono/mini/driver.c:2547
[2022/10/03 05:42:07][INFO] #34 0x0000ffff921c9900 in monovm_execute_assembly (argc=6, argv=0xaaab07c72fc0, managedAssemblyPath=<optimized out>, exitCode=0xffffc3143168) at /__w/1/s/src/mono/mono/mini/monovm.c:268
[2022/10/03 05:42:07][INFO] #35 0x0000ffff92a59554 in ?? () from /home/helixbot/work/A5400950/w/99B20887/e/performance/artifacts/bin/MicroBenchmarks/Release/net7.0/0e3e1f9c-24a4-4379-9da8-6afaf19d954f/bin/net7.0/linux-arm64/publish/libhostpolicy.so
[2022/10/03 05:42:07][INFO] #36 0x0000ffff92a59928 in ?? () from /home/helixbot/work/A5400950/w/99B20887/e/performance/artifacts/bin/MicroBenchmarks/Release/net7.0/0e3e1f9c-24a4-4379-9da8-6afaf19d954f/bin/net7.0/linux-arm64/publish/libhostpolicy.so
[2022/10/03 05:42:07][INFO] #37 0x0000ffff92a5a2f4 in corehost_main () from /home/helixbot/work/A5400950/w/99B20887/e/performance/artifacts/bin/MicroBenchmarks/Release/net7.0/0e3e1f9c-24a4-4379-9da8-6afaf19d954f/bin/net7.0/linux-arm64/publish/libhostpolicy.so
[2022/10/03 05:42:07][INFO] #38 0x0000ffff92ac7654 in ?? () from /home/helixbot/work/A5400950/w/99B20887/e/performance/artifacts/bin/MicroBenchmarks/Release/net7.0/0e3e1f9c-24a4-4379-9da8-6afaf19d954f/bin/net7.0/linux-arm64/publish/libhostfxr.so
[2022/10/03 05:42:07][INFO] #39 0x0000ffff92ac6660 in ?? () from /home/helixbot/work/A5400950/w/99B20887/e/performance/artifacts/bin/MicroBenchmarks/Release/net7.0/0e3e1f9c-24a4-4379-9da8-6afaf19d954f/bin/net7.0/linux-arm64/publish/libhostfxr.so
[2022/10/03 05:42:07][INFO] #40 0x0000ffff92ac32d4 in hostfxr_main_startupinfo () from /home/helixbot/work/A5400950/w/99B20887/e/performance/artifacts/bin/MicroBenchmarks/Release/net7.0/0e3e1f9c-24a4-4379-9da8-6afaf19d954f/bin/net7.0/linux-arm64/publish/libhostfxr.so
[2022/10/03 05:42:07][INFO] #41 0x0000aaaac92bac40 in ?? ()
[2022/10/03 05:42:07][INFO] #42 0x0000aaaac92bb004 in ?? ()
[2022/10/03 05:42:07][INFO] #43 0x0000ffff92b2ee10 in __libc_start_main (main=0xaaaac92baf74, argc=7, argv=0xffffc3143a88, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:308
[2022/10/03 05:42:07][INFO] #44 0x0000aaaac92b0034 in _start ()
[2022/10/03 05:42:07][INFO] Backtrace stopped: previous frame identical to this frame (corrupt stack?)
[2022/10/03 05:42:07][INFO] [Inferior 1 (process 86526) detached]
[2022/10/03 05:42:07][INFO] 
[2022/10/03 05:42:07][INFO] =================================================================
[2022/10/03 05:42:07][INFO] 	Basic Fault Address Reporting
[2022/10/03 05:42:07][INFO] =================================================================
[2022/10/03 05:42:07][INFO] Memory around native instruction pointer (0xffff92b41d78):0xffff92b41d68  02 00 80 d2 03 01 80 d2 e8 10 80 d2 01 00 00 d4  ................
[2022/10/03 05:42:07][INFO] 0xffff92b41d78  a5 6c 47 f9 e0 97 40 f9 a1 00 40 f9 00 00 01 eb  .lG...@...@.....
[2022/10/03 05:42:07][INFO] 0xffff92b41d88  01 00 80 d2 81 01 00 54 e0 03 04 2a f3 0b 40 f9  .......T...*..@.
[2022/10/03 05:42:07][INFO] 0xffff92b41d98  fd 7b d3 a8 c0 03 5f d6 c1 09 00 d0 21 08 47 f9  .{...._.....!.G.
[2022/10/03 05:42:07][INFO] 
[2022/10/03 05:42:07][INFO] =================================================================
[2022/10/03 05:42:07][INFO] 	Managed Stacktrace:
[2022/10/03 05:42:07][INFO] =================================================================
[2022/10/03 05:42:07][INFO] 	  at <unknown> <0xffffffff>
[2022/10/03 05:42:07][INFO] =================================================================
[2022/10/03 05:42:07][INFO] No Workload Results were obtained from the run.
@naricc
Copy link

naricc commented Oct 3, 2022

Is this causing all benchmarks to fail or just some subset?

@DrewScoggins
Copy link
Member Author

All of them are failing.

@naricc
Copy link

naricc commented Oct 4, 2022

Ok, this is because of this: dotnet/runtime#75757. We will have to use mini-jit mode instead, because llvm-jit is not available. Unfortuantely I don't think we can do that with just a command line, so I have to do the whole BDN change dance.

@fanyang-mono fanyang-mono mentioned this issue Oct 5, 2022
Merged
@naricc naricc reopened this Oct 7, 2022
@naricc
Copy link

naricc commented Oct 7, 2022

We still need to update the version number in dotnet/performance to pull the new BDN before we can close this out right?

@naricc naricc mentioned this issue Oct 7, 2022
Merged
@DrewScoggins
Copy link
Member Author

/azp run dotnet-performance

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@naricc naricc

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Mono] Disable LLVM JIT fanyang-mono/BenchmarkDotNet
Bumped BDN version naricc/performance
2 participants
@DrewScoggins @naricc