3.0.100-preview6/7/8 not working on Yocto ARM/ARM64

[chrfin]

We use .NET Core on a Variscite DART-MX8M-MINI SoM and it worked fine with 3.0.100-preview5.
With preview 6 .NET Core doesn't work at all, because as soon as a program is startet it gets stuck without any output (also nothing on the serial console) and needs to be killed. Even dotnet --info does not work.
We are running Yocto (Linux imx8mm-var-dart 4.14.78-imx8mm+g33611e8 #1 SMP PREEMPT Tue Jun 4 14:04:01 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux) on the board.

I do not know what information I can also provide, but if you need something I'm happy to provide anything you need.

[janvorli]

Would you be able to attach gdb to the hung dotnet process and then print stack trace of thread 1 and share it?

sudo gdb -p PidOfYourHungDotnet
thread select 1
bt

[chrfin]

Here the output:

root@imx8mm-var-dart:~# sudo gdb -p 636
GNU gdb (GDB) 8.0.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-poky-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 636
[New LWP 637]
[New LWP 638]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
0x0000ffffbd382320 in __lll_lock_wait (futex=futex@entry=0xffffbc61a7d8, private=0) at /usr/src/debug/glibc/2.27-r0/git/nptl/lowlevellock.c:43
43 lll_futex_wait (futex, 2, private); /* Wait if *futex == 2. /
(gdb) thread 1
[Switching to thread 1 (Thread 0xffffbcf99e80 (LWP 636))]
#0 0x0000ffffbd382320 in __lll_lock_wait (futex=futex@entry=0xffffbc61a7d8, private=0) at /usr/src/debug/glibc/2.27-r0/git/nptl/lowlevellock.c:43
43 lll_futex_wait (futex, 2, private); / Wait if *futex == 2. */
(gdb) bt
#0 0x0000ffffbd382320 in __lll_lock_wait (futex=futex@entry=0xffffbc61a7d8, private=0) at /usr/src/debug/glibc/2.27-r0/git/nptl/lowlevellock.c:43
#1 0x0000ffffbd37b648 in __GI___pthread_mutex_lock (mutex=0xffffbc61a7d8) at /usr/src/debug/glibc/2.27-r0/git/nptl/pthread_mutex_lock.c:78
#2 0x0000ffffbc5bbe68 in ust_lock_nocheck () from /usr/lib/liblttng-ust.so.0
#3 0x0000ffffbc5c17d4 in ?? () from /usr/lib/liblttng-ust.so.0
#4 0x0000ffffbc60eeb0 in ?? () from /usr/lib/liblttng-ust.so.0
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

[carlossanlop]

@janvorli should this issue be moved to another repo to increase visibility? Maybe dotnet/iot? or should it be moved to core-setup?

[chrfin]

@janvorli Just tried preview 7 with the same result :-(

[chrfin]

@janvorli @carlossanlop Still not working with preview 8 👎
Is there anything we can do to help fix this issue?

[carlossanlop]

@janvorli @joperezr @krwq who can help with this issue? Should I transfer it to another repo for better visibility?

[joperezr]

dotnet/iot is not the right place for this issue. If this should be transfered, the best place for it would be either coreclr repo or the cli repo. I would probably start with coreclr. That said, I think that either @janvorli or @jkotas would probably be the ones that could help here.

[krwq]

@chrfin how are you running dotnet on ARM64? Could you share the exact steps? Do you have TTY allocated when you are printing stuff on the console? (if not sure try running you app inside tmux which ensures TTY)

[jkotas]

This looks like duplicate of https://github.com/dotnet/coreclr/issues/15693. This hang is caused by a incompatibility between glibc and lttng in Yocto images.

To workaround it, you can either:

• Update to latest version of lttng library that has the hang fixed (https://github.com/dotnet/coreclr/issues/15693#issuecomment-479333213)
  or
• Set export COMPlus_LTTng=0 to disable use of lttng tracing in CoreCLR

[shadow-cs]

@jkotas we are facing similar issue on Yocto build of stm32mp1-dk2-mx 4.19.9 Linux on armv7. The last preview we were able to run our app on was 5 (preview 4 is working as well) every newer preview hangs before the app starts. The last line in trace is CoreCLR path = ....

Setting COMPlus_LTTng=0 does fixes the issue.

May I suggest changing the issue name to just ARM since both architectures appear to be affected.

Possibly also related to:

• dotnet/coreclr#19025
• dotnet/coreclr#24038
• dotnet/core-setup#3561

Edit: Deleting the LTTng libraries also fixes the issue (this has been discussed in other issues referenced above). Note that YOCTO might have issues building the image if LTTng is missing, but deleting it manually does work.

[chrfin]

Setting export COMPlus_LTTng=0 did work, but updating lttng-tools to 2.10.6 (previous version was 2.9.5) did not work.

[carlossanlop]

As @jkotas mentioned, this seems like a duplicate of dotnet/coreclr#15693 .
Since the workaround did not work, let's please move the discussion to that issue/repo for better visibility.