`libdl.so` not found in search path and causes a crash #53291

vadi2 · 2021-05-26T15:25:55Z

Description

Trying out the .NET 6 preview 4, I'm not able to run my application anymore due to a crash.

Configuration

.NET 6 preview 4 installed from snap
Ubuntu 20.04 LTS, x64

Regression?

Worked on .NET 5, so yes.

Other information

Running the app itself gives a segfault:

/media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/Hammer
Qt runtime not present in /media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/qt-runtime, using a default one
Segmentation fault (core dumped)

GDB shows that it's an issue with dlopen:

(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007f00526c334c in dlopen_doit (a=0x7ffecd916bd0) at dlopen.c:66
#2  0x00007f0062d8a1ef in _dl_catch_exception () from /snap/core18/current/lib/x86_64-linux-gnu/libc.so.6
#3  0x00007f0062d8a27f in _dl_catch_error () from /snap/core18/current/lib/x86_64-linux-gnu/libc.so.6
#4  0x00007f00526c3b59 in _dlerror_run (operate=operate@entry=0x7f00526c32f0 <dlopen_doit>, args=args@entry=0x7ffecd916bd0) at dlerror.c:170
#5  0x00007f00526c33da in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
#6  0x00007effe92c4895 in ?? ()
#7  0x00007ffecd916c10 in ?? ()
#8  0x00000000000b12b7 in ?? ()
#9  0x00007f0062aec3f0 in ?? () from /snap/dotnet-sdk/127/shared/Microsoft.NETCore.App/6.0.0-preview.4.21253.7/libcoreclr.so
#10 0xffffffffffffffff in ?? ()
#11 0x00007effe9899598 in ?? ()

strace shows that it's trying to find libdl.so in all places except where it actually is (/lib/x86_64-linux-gnu/libdl.so.2):

openat(AT_FDCWD, "/home/vadi/.qmlnet-qt-runtimes/qt-5.15.1-7fc8b10-linux-x64/qt/lib/preload.txt", O_RDONLY|O_CLOEXEC) = 39
fstat(39, {st_mode=S_IFREG|0664, st_size=91, ...}) = 0
flock(39, LOCK_SH|LOCK_NB)              = 0
fadvise64(39, 0, 0, POSIX_FADV_SEQUENTIAL) = 0
lseek(39, 0, SEEK_CUR)                  = 0
read(39, "libQt5QuickControls2.so.5\nlibQt5"..., 4096) = 91
read(39, "", 4096)                      = 0
flock(39, LOCK_UN)                      = 0
close(39)                               = 0
mprotect(0x7f5ac4185000, 4096, PROT_READ|PROT_WRITE) = 0
madvise(0x7f5ac4185000, 4096, MADV_DODUMP) = 0
mprotect(0x7f5ac4186000, 4096, PROT_READ|PROT_WRITE) = 0
madvise(0x7f5ac4186000, 4096, MADV_DODUMP) = 0
mprotect(0x7f5ac4179000, 4096, PROT_READ|PROT_WRITE) = 0
madvise(0x7f5ac4179000, 4096, MADV_DODUMP) = 0
mprotect(0x7f5ac3bb4000, 4096, PROT_READ|PROT_WRITE) = 0
madvise(0x7f5ac3bb4000, 4096, MADV_DODUMP) = 0
mprotect(0x7f5ac3bb4000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC) = 0
mprotect(0x7f5ac4187000, 4096, PROT_READ|PROT_WRITE) = 0
madvise(0x7f5ac4187000, 4096, MADV_DODUMP) = 0
mprotect(0x7f5ac4188000, 4096, PROT_READ|PROT_WRITE) = 0
madvise(0x7f5ac4188000, 4096, MADV_DODUMP) = 0
mprotect(0x7f5ac417a000, 4096, PROT_READ|PROT_WRITE) = 0
madvise(0x7f5ac417a000, 4096, MADV_DODUMP) = 0
mprotect(0x7f5ac4189000, 4096, PROT_READ|PROT_WRITE) = 0
madvise(0x7f5ac4189000, 4096, MADV_DODUMP) = 0
openat(AT_FDCWD, "/media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/runtimes/linux-x64/native/dl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/snap/dotnet-sdk/127/shared/Microsoft.NETCore.App/6.0.0-preview.4.21253.7/dl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/dl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/snap/dotnet-sdk/127/shared/Microsoft.NETCore.App/6.0.0-preview.4.21253.7/../../../usr/lib/x86_64-linux-gnu/dl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/snap/dotnet-sdk/127/shared/Microsoft.NETCore.App/6.0.0-preview.4.21253.7/../../../lib/x86_64-linux-gnu/dl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/snap/core18/current/lib/x86_64-linux-gnu/dl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/snap/core18/current/lib/x86_64-linux-gnu/dl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 39
fstat(39, {st_mode=S_IFREG|0644, st_size=113889, ...}) = 0
mmap(NULL, 113889, PROT_READ, MAP_PRIVATE, 39, 0) = 0x7f5b33bce000
close(39)                               = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/dl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/dl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/dl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/dl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
munmap(0x7f5b33bce000, 113889)          = 0
openat(AT_FDCWD, "/media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/runtimes/linux-x64/native/libdl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/snap/dotnet-sdk/127/shared/Microsoft.NETCore.App/6.0.0-preview.4.21253.7/libdl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/libdl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/snap/dotnet-sdk/127/shared/Microsoft.NETCore.App/6.0.0-preview.4.21253.7/../../../usr/lib/x86_64-linux-gnu/libdl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/snap/dotnet-sdk/127/shared/Microsoft.NETCore.App/6.0.0-preview.4.21253.7/../../../lib/x86_64-linux-gnu/libdl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/snap/core18/current/lib/x86_64-linux-gnu/libdl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/snap/core18/current/lib/x86_64-linux-gnu/libdl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 39
fstat(39, {st_mode=S_IFREG|0644, st_size=113889, ...}) = 0
mmap(NULL, 113889, PROT_READ, MAP_PRIVATE, 39, 0) = 0x7f5b33bce000
close(39)                               = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libdl.so", O_RDONLY|O_CLOEXEC) = 39
read(39, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \22\0\0\0\0\0\0"..., 832) = 832
fstat(39, {st_mode=S_IFREG|0644, st_size=18816, ...}) = 0
mmap(NULL, 20752, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 39, 0) = 0x7f5b33bc8000
mmap(0x7f5b33bc9000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 39, 0x1000) = 0x7f5b33bc9000
mmap(0x7f5b33bcb000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 39, 0x3000) = 0x7f5b33bcb000
mmap(0x7f5b33bcc000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 39, 0x3000) = 0x7f5b33bcc000
close(39)                               = 0
mprotect(0x7f5b33bcc000, 4096, PROT_READ) = 0
munmap(0x7f5b33bce000, 113889)          = 0
futex(0x7f5b33bcd0c8, FUTEX_WAKE_PRIVATE, 2147483647) = 0
mprotect(0x7f5ac417b000, 4096, PROT_READ|PROT_WRITE) = 0
madvise(0x7f5ac417b000, 4096, MADV_DODUMP) = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=NULL} ---
rt_sigprocmask(SIG_UNBLOCK, [RT_2], NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [SEGV], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [RT_2], NULL, 8) = 0
rt_sigaction(SIGSEGV, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f5b3e0a4980}, NULL, 8) = 0
unlink("/tmp/clr-debug-pipe-4901-4339-in") = 0
unlink("/tmp/clr-debug-pipe-4901-4339-out") = 0
unlink("/tmp/dotnet-diagnostic-4901-4339-socket") = 0
rt_sigreturn({mask=[]})                 = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=NULL} ---
+++ killed by SIGSEGV (core dumped) +++
Segmentation fault (core dumped)

How to fix?

The text was updated successfully, but these errors were encountered:

ghost · 2021-05-26T15:26:00Z

Tagging subscribers to this area: @carlossanlop
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

Trying out the .NET 6 preview 4, I'm not able to run my application anymore due to a crash.

Configuration

.NET 6 preview 4 installed from snap
Ubuntu 20.04 LTS, x64

Regression?

Worked on .NET 5, so yes.

Other information

Running the app itself gives a segfault:

/media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/Hammer
Qt runtime not present in /media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/qt-runtime, using a default one
Segmentation fault (core dumped)

GDB shows that it's an issue with dlopen:

(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007f00526c334c in dlopen_doit (a=0x7ffecd916bd0) at dlopen.c:66
#2  0x00007f0062d8a1ef in _dl_catch_exception () from /snap/core18/current/lib/x86_64-linux-gnu/libc.so.6
#3  0x00007f0062d8a27f in _dl_catch_error () from /snap/core18/current/lib/x86_64-linux-gnu/libc.so.6
#4  0x00007f00526c3b59 in _dlerror_run (operate=operate@entry=0x7f00526c32f0 <dlopen_doit>, args=args@entry=0x7ffecd916bd0) at dlerror.c:170
#5  0x00007f00526c33da in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
#6  0x00007effe92c4895 in ?? ()
#7  0x00007ffecd916c10 in ?? ()
#8  0x00000000000b12b7 in ?? ()
#9  0x00007f0062aec3f0 in ?? () from /snap/dotnet-sdk/127/shared/Microsoft.NETCore.App/6.0.0-preview.4.21253.7/libcoreclr.so
#10 0xffffffffffffffff in ?? ()
#11 0x00007effe9899598 in ?? ()

strace shows that it's trying to find libdl.so in all places except where it actually is (/lib/x86_64-linux-gnu/libdl.so.2):

openat(AT_FDCWD, "/home/vadi/.qmlnet-qt-runtimes/qt-5.15.1-7fc8b10-linux-x64/qt/lib/preload.txt", O_RDONLY|O_CLOEXEC) = 39
fstat(39, {st_mode=S_IFREG|0664, st_size=91, ...}) = 0
flock(39, LOCK_SH|LOCK_NB)              = 0
fadvise64(39, 0, 0, POSIX_FADV_SEQUENTIAL) = 0
lseek(39, 0, SEEK_CUR)                  = 0
read(39, "libQt5QuickControls2.so.5\nlibQt5"..., 4096) = 91
read(39, "", 4096)                      = 0
flock(39, LOCK_UN)                      = 0
close(39)                               = 0
mprotect(0x7f5ac4185000, 4096, PROT_READ|PROT_WRITE) = 0
madvise(0x7f5ac4185000, 4096, MADV_DODUMP) = 0
mprotect(0x7f5ac4186000, 4096, PROT_READ|PROT_WRITE) = 0
madvise(0x7f5ac4186000, 4096, MADV_DODUMP) = 0
mprotect(0x7f5ac4179000, 4096, PROT_READ|PROT_WRITE) = 0
madvise(0x7f5ac4179000, 4096, MADV_DODUMP) = 0
mprotect(0x7f5ac3bb4000, 4096, PROT_READ|PROT_WRITE) = 0
madvise(0x7f5ac3bb4000, 4096, MADV_DODUMP) = 0
mprotect(0x7f5ac3bb4000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC) = 0
mprotect(0x7f5ac4187000, 4096, PROT_READ|PROT_WRITE) = 0
madvise(0x7f5ac4187000, 4096, MADV_DODUMP) = 0
mprotect(0x7f5ac4188000, 4096, PROT_READ|PROT_WRITE) = 0
madvise(0x7f5ac4188000, 4096, MADV_DODUMP) = 0
mprotect(0x7f5ac417a000, 4096, PROT_READ|PROT_WRITE) = 0
madvise(0x7f5ac417a000, 4096, MADV_DODUMP) = 0
mprotect(0x7f5ac4189000, 4096, PROT_READ|PROT_WRITE) = 0
madvise(0x7f5ac4189000, 4096, MADV_DODUMP) = 0
openat(AT_FDCWD, "/media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/runtimes/linux-x64/native/dl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/snap/dotnet-sdk/127/shared/Microsoft.NETCore.App/6.0.0-preview.4.21253.7/dl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/dl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/snap/dotnet-sdk/127/shared/Microsoft.NETCore.App/6.0.0-preview.4.21253.7/../../../usr/lib/x86_64-linux-gnu/dl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/snap/dotnet-sdk/127/shared/Microsoft.NETCore.App/6.0.0-preview.4.21253.7/../../../lib/x86_64-linux-gnu/dl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/snap/core18/current/lib/x86_64-linux-gnu/dl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/snap/core18/current/lib/x86_64-linux-gnu/dl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 39
fstat(39, {st_mode=S_IFREG|0644, st_size=113889, ...}) = 0
mmap(NULL, 113889, PROT_READ, MAP_PRIVATE, 39, 0) = 0x7f5b33bce000
close(39)                               = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/dl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/dl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/dl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/dl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
munmap(0x7f5b33bce000, 113889)          = 0
openat(AT_FDCWD, "/media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/runtimes/linux-x64/native/libdl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/snap/dotnet-sdk/127/shared/Microsoft.NETCore.App/6.0.0-preview.4.21253.7/libdl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/libdl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/snap/dotnet-sdk/127/shared/Microsoft.NETCore.App/6.0.0-preview.4.21253.7/../../../usr/lib/x86_64-linux-gnu/libdl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/snap/dotnet-sdk/127/shared/Microsoft.NETCore.App/6.0.0-preview.4.21253.7/../../../lib/x86_64-linux-gnu/libdl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/snap/core18/current/lib/x86_64-linux-gnu/libdl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/snap/core18/current/lib/x86_64-linux-gnu/libdl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 39
fstat(39, {st_mode=S_IFREG|0644, st_size=113889, ...}) = 0
mmap(NULL, 113889, PROT_READ, MAP_PRIVATE, 39, 0) = 0x7f5b33bce000
close(39)                               = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libdl.so", O_RDONLY|O_CLOEXEC) = 39
read(39, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \22\0\0\0\0\0\0"..., 832) = 832
fstat(39, {st_mode=S_IFREG|0644, st_size=18816, ...}) = 0
mmap(NULL, 20752, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 39, 0) = 0x7f5b33bc8000
mmap(0x7f5b33bc9000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 39, 0x1000) = 0x7f5b33bc9000
mmap(0x7f5b33bcb000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 39, 0x3000) = 0x7f5b33bcb000
mmap(0x7f5b33bcc000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 39, 0x3000) = 0x7f5b33bcc000
close(39)                               = 0
mprotect(0x7f5b33bcc000, 4096, PROT_READ) = 0
munmap(0x7f5b33bce000, 113889)          = 0
futex(0x7f5b33bcd0c8, FUTEX_WAKE_PRIVATE, 2147483647) = 0
mprotect(0x7f5ac417b000, 4096, PROT_READ|PROT_WRITE) = 0
madvise(0x7f5ac417b000, 4096, MADV_DODUMP) = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=NULL} ---
rt_sigprocmask(SIG_UNBLOCK, [RT_2], NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [SEGV], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [RT_2], NULL, 8) = 0
rt_sigaction(SIGSEGV, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f5b3e0a4980}, NULL, 8) = 0
unlink("/tmp/clr-debug-pipe-4901-4339-in") = 0
unlink("/tmp/clr-debug-pipe-4901-4339-out") = 0
unlink("/tmp/dotnet-diagnostic-4901-4339-socket") = 0
rt_sigreturn({mask=[]})                 = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=NULL} ---
+++ killed by SIGSEGV (core dumped) +++
Segmentation fault (core dumped)

How to fix?

Author:	vadi2
Assignees:	-
Labels:	`area-System.IO`, `untriaged`
Milestone:	-

janvorli · 2021-06-01T14:15:53Z

Looking at the dump above, I can see that it has succeeded loading the libdl:
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libdl.so", O_RDONLY|O_CLOEXEC) = 39
It looks like the problem is elsewhere. @vadi2 would you be able to run it under lldb and get a stack trace so that the location where the SIGSEGV happened can be identified?
If you are not familiar with lldb, the stack trace command is bt. If there are any private details in the stack trace, please feel free to remove them. But it looks like this is crashing at an early initialization time, so I guess only .NET frames would be on the stack.

vadi2 · 2021-06-01T16:14:09Z

You're right, thanks for spotting that libdl.so does load.

lldb shows the following:

✦ ❯ lldb /media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/Hammer
(lldb) target create "/media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/Hammer"
Current executable set to '/media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/Hammer' (x86_64).
(lldb) run
Process 83804 launched: '/media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/Hammer' (x86_64)
Qt runtime not present in /media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/qt-runtime, using a default one
Process 83804 stopped
* thread #1, name = 'Hammer', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
    frame #0: 0x0000000000000000
error: memory read failed for 0x0
(lldb) bt
* thread #1, name = 'Hammer', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
  * frame #0: 0x0000000000000000
    frame #1: 0x00007ffff42a834c libdl.so`___lldb_unnamed_symbol5$$libdl.so + 92
    frame #2: 0x00007ffff6f8f1ef libc.so.6`_dl_catch_exception + 111
    frame #3: 0x00007ffff6f8f27f libc.so.6`_dl_catch_error + 47
    frame #4: 0x00007ffff42a8b59 libdl.so`___lldb_unnamed_symbol12$$libdl.so + 105
    frame #5: 0x00007ffff42a83da libdl.so`dlopen + 74
    frame #6: 0x00007fff7d5b4895
    frame #7: 0x00007fff7d5b475b
    frame #8: 0x00007fff7d5b4708
    frame #9: 0x00007fff7d5b43d2
    frame #10: 0x00007fff7d5b4334
    frame #11: 0x00007fff7d5b42e8
    frame #12: 0x00007fff7d5b2b2c
    frame #13: 0x00007fff7d5b0d05
    frame #14: 0x00007fff7d5a0c79
    frame #15: 0x00007ffff6a7b3e7 libcoreclr.so`___lldb_unnamed_symbol9554$$libcoreclr.so + 124
    frame #16: 0x00007ffff68cf63b libcoreclr.so`___lldb_unnamed_symbol4301$$libcoreclr.so + 1643
    frame #17: 0x00007ffff67a1d0b libcoreclr.so`___lldb_unnamed_symbol298$$libcoreclr.so + 923
    frame #18: 0x00007ffff67a2039 libcoreclr.so`___lldb_unnamed_symbol299$$libcoreclr.so + 393
    frame #19: 0x00007ffff67e1423 libcoreclr.so`___lldb_unnamed_symbol1150$$libcoreclr.so + 627
    frame #20: 0x00007ffff678a90d libcoreclr.so`coreclr_execute_assembly + 413
    frame #21: 0x00007ffff7f216e1 libhostpolicy.so`___lldb_unnamed_symbol160$$libhostpolicy.so + 1345
    frame #22: 0x00007ffff7f21b71 libhostpolicy.so`___lldb_unnamed_symbol161$$libhostpolicy.so + 49
    frame #23: 0x00007ffff7f2259c libhostpolicy.so`corehost_main + 172
    frame #24: 0x00007ffff7f7e2a9 libhostfxr.so`___lldb_unnamed_symbol95$$libhostfxr.so + 1817
    frame #25: 0x00007ffff7f7c9ab libhostfxr.so`___lldb_unnamed_symbol93$$libhostfxr.so + 667
    frame #26: 0x00007ffff7f77e9b libhostfxr.so`hostfxr_main_startupinfo + 171
    frame #27: 0x0000555555564545 Hammer`___lldb_unnamed_symbol152$$Hammer + 1045
    frame #28: 0x0000555555564a10 Hammer`___lldb_unnamed_symbol153$$Hammer + 144
    frame #29: 0x00007ffff6e49bf7 libc.so.6`__libc_start_main + 231
    frame #30: 0x0000555555558d8a Hammer`___lldb_unnamed_symbol33$$Hammer + 41
(lldb)

janvorli · 2021-06-01T17:36:39Z

We will need to get symbols for the libdl.so and libc6 to see more. Can you please install libc6-dbg Ubuntu package and try again? lldb should see the symbols then. If not, we may need to point it to the debug symbols directory.

vadi2 · 2021-06-01T18:12:03Z

It seems it's already installed. How to point it to the symbols?

✦ ❯ sudo apt install libc6-dbg
[sudo] password for vadi: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
libc6-dbg is already the newest version (2.31-0ubuntu9.2).
libc6-dbg set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 16 not upgraded.

~ took 2s 
✦ ❯ lldb /media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/Hammer
(lldb) target create "/media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/Hammer"
Current executable set to '/media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/Hammer' (x86_64).
(lldb) run
Process 194011 launched: '/media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/Hammer' (x86_64)
Qt runtime not present in /media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/qt-runtime, using a default one
Process 194011 stopped
* thread #1, name = 'Hammer', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
    frame #0: 0x0000000000000000
error: memory read failed for 0x0
(lldb) bt
* thread #1, name = 'Hammer', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
  * frame #0: 0x0000000000000000
    frame #1: 0x00007ffff42a834c libdl.so`___lldb_unnamed_symbol5$$libdl.so + 92
    frame #2: 0x00007ffff6f8f1ef libc.so.6`_dl_catch_exception + 111
    frame #3: 0x00007ffff6f8f27f libc.so.6`_dl_catch_error + 47
    frame #4: 0x00007ffff42a8b59 libdl.so`___lldb_unnamed_symbol12$$libdl.so + 105
    frame #5: 0x00007ffff42a83da libdl.so`dlopen + 74
    frame #6: 0x00007fff7d5c4895
    frame #7: 0x00007fff7d5c475b
    frame #8: 0x00007fff7d5c4708
    frame #9: 0x00007fff7d5c43d2
    frame #10: 0x00007fff7d5c4334
    frame #11: 0x00007fff7d5c42e8
    frame #12: 0x00007fff7d5c2b2c
    frame #13: 0x00007fff7d5c0d05
    frame #14: 0x00007fff7d5b0c79
    frame #15: 0x00007ffff6a7b3e7 libcoreclr.so`___lldb_unnamed_symbol9554$$libcoreclr.so + 124
    frame #16: 0x00007ffff68cf63b libcoreclr.so`___lldb_unnamed_symbol4301$$libcoreclr.so + 1643
    frame #17: 0x00007ffff67a1d0b libcoreclr.so`___lldb_unnamed_symbol298$$libcoreclr.so + 923
    frame #18: 0x00007ffff67a2039 libcoreclr.so`___lldb_unnamed_symbol299$$libcoreclr.so + 393
    frame #19: 0x00007ffff67e1423 libcoreclr.so`___lldb_unnamed_symbol1150$$libcoreclr.so + 627
    frame #20: 0x00007ffff678a90d libcoreclr.so`coreclr_execute_assembly + 413
    frame #21: 0x00007ffff7f216e1 libhostpolicy.so`___lldb_unnamed_symbol160$$libhostpolicy.so + 1345
    frame #22: 0x00007ffff7f21b71 libhostpolicy.so`___lldb_unnamed_symbol161$$libhostpolicy.so + 49
    frame #23: 0x00007ffff7f2259c libhostpolicy.so`corehost_main + 172
    frame #24: 0x00007ffff7f7e2a9 libhostfxr.so`___lldb_unnamed_symbol95$$libhostfxr.so + 1817
    frame #25: 0x00007ffff7f7c9ab libhostfxr.so`___lldb_unnamed_symbol93$$libhostfxr.so + 667
    frame #26: 0x00007ffff7f77e9b libhostfxr.so`hostfxr_main_startupinfo + 171
    frame #27: 0x0000555555564545 Hammer`___lldb_unnamed_symbol152$$Hammer + 1045
    frame #28: 0x0000555555564a10 Hammer`___lldb_unnamed_symbol153$$Hammer + 144
    frame #29: 0x00007ffff6e49bf7 libc.so.6`__libc_start_main + 231
    frame #30: 0x0000555555558d8a Hammer`___lldb_unnamed_symbol33$$Hammer + 41
(lldb)

janvorli · 2021-06-01T18:36:17Z

can you please try the following lldb commands before executing the bt?

target symbols add /usr/lib/debug/lib/x86_64-linux-gnu/libdl-2.31.so
target symbols add /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.31.so

vadi2 · 2021-06-02T09:37:36Z

Sure, here it is:

✦ ❯ lldb /media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/Hammer
(lldb) target create "/media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/Hammer"
Current executable set to '/media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/Hammer' (x86_64).
(lldb) run
Process 416270 launched: '/media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/Hammer' (x86_64)
Qt runtime not present in /media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/qt-runtime, using a default one
Process 416270 stopped
* thread #1, name = 'Hammer', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
    frame #0: 0x0000000000000000
error: memory read failed for 0x0
(lldb) target symbols add /usr/lib/debug/lib/x86_64-linux-gnu/libdl-2.31.so
symbol file '/usr/lib/debug/lib/x86_64-linux-gnu/libdl-2.31.so' has been added to '/lib/x86_64-linux-gnu/libdl.so'
(lldb) target symbols add /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.31.so
error: symbol file '/usr/lib/debug/lib/x86_64-linux-gnu/libc-2.31.so' does not match any existing module
(lldb) bt
* thread #1, name = 'Hammer', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
  * frame #0: 0x0000000000000000
    frame #1: 0x00007ffff42a834c libdl.so`dlopen_doit(a=0x00007fffffffca80) at dlopen.c:66:15
    frame #2: 0x00007ffff6f8f1ef libc.so.6`_dl_catch_exception + 111
    frame #3: 0x00007ffff6f8f27f libc.so.6`_dl_catch_error + 47
    frame #4: 0x00007ffff42a8b59 libdl.so`_dlerror_run(operate=(libdl.so`dlopen_doit at dlopen.c:58:1), args=0x00007fffffffca80) at dlerror.c:170:21
    frame #5: 0x00007ffff42a83da libdl.so`__dlopen(file=<unavailable>, mode=<unavailable>) at dlopen.c:87:10
    frame #6: 0x00007fff7d5a4895
    frame #7: 0x00007fff7d5a475b
    frame #8: 0x00007fff7d5a4708
    frame #9: 0x00007fff7d5a43d2
    frame #10: 0x00007fff7d5a4334
    frame #11: 0x00007fff7d5a42e8
    frame #12: 0x00007fff7d5a2b2c
    frame #13: 0x00007fff7d5a0d05
    frame #14: 0x00007fff7d590c79
    frame #15: 0x00007ffff6a7b3e7 libcoreclr.so`___lldb_unnamed_symbol9554$$libcoreclr.so + 124
    frame #16: 0x00007ffff68cf63b libcoreclr.so`___lldb_unnamed_symbol4301$$libcoreclr.so + 1643
    frame #17: 0x00007ffff67a1d0b libcoreclr.so`___lldb_unnamed_symbol298$$libcoreclr.so + 923
    frame #18: 0x00007ffff67a2039 libcoreclr.so`___lldb_unnamed_symbol299$$libcoreclr.so + 393
    frame #19: 0x00007ffff67e1423 libcoreclr.so`___lldb_unnamed_symbol1150$$libcoreclr.so + 627
    frame #20: 0x00007ffff678a90d libcoreclr.so`coreclr_execute_assembly + 413
    frame #21: 0x00007ffff7f216e1 libhostpolicy.so`___lldb_unnamed_symbol160$$libhostpolicy.so + 1345
    frame #22: 0x00007ffff7f21b71 libhostpolicy.so`___lldb_unnamed_symbol161$$libhostpolicy.so + 49
    frame #23: 0x00007ffff7f2259c libhostpolicy.so`corehost_main + 172
    frame #24: 0x00007ffff7f7e2a9 libhostfxr.so`___lldb_unnamed_symbol95$$libhostfxr.so + 1817
    frame #25: 0x00007ffff7f7c9ab libhostfxr.so`___lldb_unnamed_symbol93$$libhostfxr.so + 667
    frame #26: 0x00007ffff7f77e9b libhostfxr.so`hostfxr_main_startupinfo + 171
    frame #27: 0x0000555555564545 Hammer`___lldb_unnamed_symbol152$$Hammer + 1045
    frame #28: 0x0000555555564a10 Hammer`___lldb_unnamed_symbol153$$Hammer + 144
    frame #29: 0x00007ffff6e49bf7 libc.so.6`__libc_start_main + 231
    frame #30: 0x0000555555558d8a Hammer`___lldb_unnamed_symbol33$$Hammer + 41
(lldb)

janvorli · 2021-06-02T09:44:44Z

Ok, great. Can you please run these lldb commands?

f 1
disass -a $rip

vadi2 · 2021-06-02T09:50:41Z

(lldb) f 1
frame #1: 0x00007ffff42a834c libdl.so`dlopen_doit(a=0x00007fffffffca80) at dlopen.c:66:15
(lldb) disass -a $rip
libdl.so`dlopen_doit:
    0x7ffff42a82f0 <+0>:   endbr64 
    0x7ffff42a82f4 <+4>:   pushq  %rbx
    0x7ffff42a82f5 <+5>:   movl   0x8(%rdi), %esi
    0x7ffff42a82f8 <+8>:   testl  $0xbfffeef0, %esi         ; imm = 0xBFFFEEF0 
    0x7ffff42a82fe <+14>:  jne    0x7ffff42a8364            ; <+116> [inlined] dlopen_doit at dlopen.c:57
    0x7ffff42a8300 <+16>:  movq   0x3cd1(%rip), %rdx
    0x7ffff42a8307 <+23>:  movq   %rdi, %rbx
    0x7ffff42a830a <+26>:  movq   0x3caf(%rip), %rax
    0x7ffff42a8311 <+33>:  orl    $0x80000000, %esi         ; imm = 0x80000000 
    0x7ffff42a8317 <+39>:  movq   (%rdi), %rdi
    0x7ffff42a831a <+42>:  movq   0x3ddf(%rip), %r9         ; __dlfcn_argv
    0x7ffff42a8321 <+49>:  movq   $-0x2, %r10
    0x7ffff42a8328 <+56>:  movq   (%rdx), %rcx
    0x7ffff42a832b <+59>:  movq   0x1e8(%rax), %rax
    0x7ffff42a8332 <+66>:  movl   0x3dcf(%rip), %r8d        ; __dlfcn_argc
    0x7ffff42a8339 <+73>:  movq   0x18(%rbx), %rdx
    0x7ffff42a833d <+77>:  testq  %rdi, %rdi
    0x7ffff42a8340 <+80>:  je     0x7ffff42a8358            ; <+104> at dlopen.c:66:15
    0x7ffff42a8342 <+82>:  subq   $0x8, %rsp
    0x7ffff42a8346 <+86>:  pushq  %rcx
    0x7ffff42a8347 <+87>:  movq   %r10, %rcx
    0x7ffff42a834a <+90>:  callq  *%rax
->  0x7ffff42a834c <+92>:  movq   %rax, 0x10(%rbx)
    0x7ffff42a8350 <+96>:  popq   %rax
    0x7ffff42a8351 <+97>:  popq   %rdx
    0x7ffff42a8352 <+98>:  popq   %rbx
    0x7ffff42a8353 <+99>:  retq   
    0x7ffff42a8354 <+100>: nopl   (%rax)
    0x7ffff42a8358 <+104>: xorl   %r10d, %r10d
    0x7ffff42a835b <+107>: leaq   0x1cc5(%rip), %rdi
    0x7ffff42a8362 <+114>: jmp    0x7ffff42a8342            ; <+82> at dlopen.c:66:15
    0x7ffff42a8364 <+116>: movq   0x3c65(%rip), %rdi
    0x7ffff42a836b <+123>: movl   $0x5, %edx
    0x7ffff42a8370 <+128>: leaq   0x1c89(%rip), %rsi
    0x7ffff42a8377 <+135>: callq  0x7ffff42a81b0
    0x7ffff42a837c <+140>: xorl   %edx, %edx
    0x7ffff42a837e <+142>: xorl   %esi, %esi
    0x7ffff42a8380 <+144>: xorl   %edi, %edi
    0x7ffff42a8382 <+146>: movq   %rax, %rcx
    0x7ffff42a8385 <+149>: callq  0x7ffff42a81c0
(lldb)

janvorli · 2021-06-02T09:58:32Z

Thank you, now the following please:

x/gx 0x7ffff42a8311+0x3caf

janvorli · 2021-06-02T09:59:36Z

(I am trying to find out where the callq *rax would go)

vadi2 · 2021-06-02T10:03:52Z

Yep sure - it says:

(lldb) x/gx 0x7ffff42a8311+0x3caf
0x7ffff42abfc0: 0x00007ffff7ffc760
(lldb)

janvorli · 2021-06-02T10:06:00Z

Ok, so now:

x/gx 0x00007ffff7ffc760+0x1e8

vadi2 · 2021-06-02T10:06:56Z

(lldb) x/gx 0x7ffff42a8311+0x3caf
0x7ffff42abfc0: 0x00007ffff7ffc760
(lldb) x/gx 0x00007ffff7ffc760+0x1e8
0x7ffff7ffc948: 0x0000000000000000
(lldb)

janvorli · 2021-06-02T10:20:35Z

Ok, here we go, that's the address it would call and it is really zero. It looks like the code is this:
https://code.woboq.org/userspace/glibc/dlfcn/dlopen.c.html#66
That would mean that the _dl_open symbol was not found and thus it is set to NULL. I don't have any idea on why would that happen as it is out of .NET control, maybe something snap related.
Can you please try:

p/x _dl_open

vadi2 · 2021-06-02T10:21:45Z

Hmm:

(lldb) p/x _dl_open
error: <user expression 0>:1:1: use of undeclared identifier '_dl_open'
_dl_open
^

vadi2 · 2021-06-02T10:21:59Z

If it helps, my project is open source located @ https://github.com/health-validator/Hammer/tree/test-net6.0.

janvorli · 2021-06-02T10:24:24Z

Ok, great, I can try to check it myself. I suspect snap being related to the issue.
One more thing, can you please run the following? The symbol should be in ld-2.31.so and I'd like to see where it comes from.

target modules list

vadi2 · 2021-06-02T11:18:15Z

For certain, here it is:

(lldb) target modules list
[  0] 01744AC3-EDBF-7E7A-47AA-2A069ED0C9BE-4457DF00 0x0000555555554000 /media/vadi/SSDer/Programs/Hammer/bin/Debug/net6.0/Hammer 
[  1] 977C39FE-87AB-FA42-6D30-43F6C8E21F7B-E3F0E876 0x00007ffff7dd3000 /snap/core18/2066/lib/x86_64-linux-gnu/ld-2.27.so 
[  2] 694EFEA9-7CEF-DA51-B158-768AD7768F5B-A8F34EA8 0x00007ffff7ffa000 [vdso] (0x00007ffff7ffa000)
[  3] 68F36706-EB2E-6EEE-4046-C4FDCA2A1954-0B2F6113 0x00007ffff7bb4000 /snap/core18/current/lib/x86_64-linux-gnu/libpthread.so.0 
[  4] C5396CFC-F808-B313-7C70-C50F87A21BB9-CEBD4824 0x00007ffff79b0000 /snap/core18/current/lib/x86_64-linux-gnu/libdl.so.2 
[  5] C4581310-7D14-87F8-FC71-97424AA6F1E0-BE00FE6C 0x00007ffff77cf000 /lib/x86_64-linux-gnu/libstdc++.so.6 
[  6] 39ED4F82-6411-9C71-15DF-214E0A34DE90-AACDBE77 0x00007ffff7431000 /snap/core18/current/lib/x86_64-linux-gnu/libm.so.6 
[  7] 679F3AE1-1120-EC7C-483B-C9295345D836-F5C104F7 0x00007ffff7219000 /snap/core18/current/lib/x86_64-linux-gnu/libgcc_s.so.1 
[  8] CE450EB0-1A5E-5ACC-7CE7-B8C2633B02CC-1093339E 0x00007ffff6e28000 /snap/core18/current/lib/x86_64-linux-gnu/libc.so.6 
[  9] 3E0CF3D4-2789-2F63-97A4-46B279E1E513-6060F38B 0x00007ffff7f6a000 /snap/dotnet-sdk/127/host/fxr/6.0.0-preview.4.21253.7/libhostfxr.so 
[ 10] 8EF21BDD-6704-F9C7-27EC-4353B342BC48-3CE184A6 0x00007ffff7f09000 /snap/dotnet-sdk/127/shared/Microsoft.NETCore.App/6.0.0-preview.4.21253.7/libhostpolicy.so 
[ 11] 5CFD8191-620B-81BC-614F-2312E6268879-8DA8F01C 0x00007ffff6716000 /snap/dotnet-sdk/127/shared/Microsoft.NETCore.App/6.0.0-preview.4.21253.7/libcoreclr.so 
[ 12] 8BB230E1-D019-753B-0193-13B61A920224-F1A372C9 0x00007ffff650e000 /snap/core18/current/lib/x86_64-linux-gnu/librt.so.1 
[ 13] 505897EA-90EC-0083-E3C2-C3401E80FB3A-985DB801 0x00007ffff7e5f000 /snap/dotnet-sdk/127/shared/Microsoft.NETCore.App/6.0.0-preview.4.21253.7/libcoreclrtraceptprovider.so 
[ 14] 007E931A-0024-2C46-ADD5-25C4E2CAE2A7-94BA20E7 0x00007ffff628f000 /snap/dotnet-sdk/127/usr/lib/x86_64-linux-gnu/liblttng-ust.so.0 
[ 15] 453D3F91-E0BD-234B-8E71-74864ADC0C07-D18F4717 0x00007ffff6072000 /snap/dotnet-sdk/127/usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0 
[ 16] 30E15DB0-85CA-D1BF-049C-5792D8310854-4A25327A 0x00007ffff5e69000 /snap/dotnet-sdk/127/usr/lib/x86_64-linux-gnu/liburcu-bp.so.6 
[ 17] A6FC4348-438F-3286-793D-841119767F87-F94CD238 0x00007ffff5c5e000 /snap/dotnet-sdk/127/usr/lib/x86_64-linux-gnu/liburcu-cds.so.6 
[ 18] 836D1645-4659-5D11-7A0B-32F171160C98-B2E02DE0 0x00007fff7c34e000 /snap/dotnet-sdk/127/shared/Microsoft.NETCore.App/6.0.0-preview.4.21253.7/libclrjit.so 
[ 19] FB0B616B-D774-415B-CD17-5E0313B5BB33-0CD5FDB4 0x00007ffff42ce000 /snap/dotnet-sdk/127/shared/Microsoft.NETCore.App/6.0.0-preview.4.21253.7/libSystem.Native.so 
[ 20] 3CABDDE9-BB67-44AA-C535-FABB11399D4D-4E72F932 0x00007fff4ec1a000 /snap/dotnet-sdk/127/usr/lib/x86_64-linux-gnu/libicuuc.so.60 
[ 21] C4974FC9-FA56-7DE6-E95C-F1364706B3CF-79175FB5 0x00007fff4d071000 /snap/dotnet-sdk/127/usr/lib/x86_64-linux-gnu/libicudata.so.60 
[ 22] C74FCD0F-E314-58E1-4A2F-FE21183023F5-81ACD388 0x00007fff4cb5b000 /snap/dotnet-sdk/127/usr/lib/x86_64-linux-gnu/libicui18n.so.60 
[ 23] A3F78028-221C-F3B7-84DD-5E10D05AD7A5-5A6A85D5 0x00007ffff42a7000 /lib/x86_64-linux-gnu/libdl.so 
(lldb)

janvorli · 2021-06-02T11:31:40Z

I can see that it uses /snap/core18/2066/lib/x86_64-linux-gnu/ld-2.27.so. It is likely that it is not compatible with the /lib/x86_64-linux-gnu/libdl.so that comes from the OS.
Interestingly, I can also see /snap/core18/current/lib/x86_64-linux-gnu/libdl.so.2 loaded in the process. So it seems like a problem that the app or runtime is trying to pinvoke into the libdl.so instead of libdl.so.2. I'll check locally what's calling it.

vadi2 · 2021-06-02T12:04:27Z

Hmm. I'm using https://github.com/qmlnet/qmlnet which I'm not very familiar with, but I know that it does a few things differently under the hood in order to deliver better performance and avoid pinvoke all the time.

janvorli · 2021-06-02T13:40:23Z

I believe that's the problem. See the comment in this section:
https://github.com/qmlnet/qmlnet/blob/0836ae783a3ed5fb404db68d0706f9a7fa3765b4/README.md#getting-started
This is wrong, as the libdl.so is meant for compile time linking only. At runtime, applications should pinvoke to libdl.so.2 instead.
The libc6-dev package (as any -dev package) is supposed to be used only when compiling applications, not as a runtime dependency.
When loading a library via dlopen using a name, a SONAME of that library should be used. For libdl, the soname is:

 objdump -p /lib/x86_64-linux-gnu/libdl.so.2 | grep SONAME
  SONAME               libdl.so.2

notice that you get the same soname for the libdl.so:

objdump -p /usr/lib/x86_64-linux-gnu/libdl.so | grep SONAME
  SONAME               libdl.so.2

this is because the libdl.so is just a symlink to the libdl.so.2

ls -la /usr/lib/x86_64-linux-gnu/libdl.so
lrwxrwxrwx 1 root root 32 dub 21 11:14 /usr/lib/x86_64-linux-gnu/libdl.so -> /lib/x86_64-linux-gnu/libdl.so.2

However, the strange thing is that the qmlnet seems to refer to the libdl.so only here:
https://github.com/qmlnet/qmlnet/blob/0836ae783a3ed5fb404db68d0706f9a7fa3765b4/src/net/Qml.Net/Internal/MacDllImportLibraryPathResolver.cs#L69-L76

Which is a macOS stuff.

janvorli · 2021-06-02T14:11:18Z

I am testing it with a dotnet installed without snap, but it seems that the issue actually comes from here:
https://github.com/pauldotknopf/net-native-lib-loader/blob/344a1df38aa6ec0d827c972375c6e2d7eac3d3b9/src/NetNativeLibLoader/Loader/Native/dl.cs#L31
This is the stack trace with managed frames that matches your failure point:

OS Thread Id: 0x7f73 (1)
        Child SP               IP Call Site
00007FFFFFFFD7C0 00007FFFF79B6F74 libdl.so.2!dlopen@@GLIBC_2.2.5 + 4 at /build/glibc-S7Ft5T/glibc-2.23/dlfcn/dlopen.c:77
00007FFFFFFFD7F0 00007FFF7D4EE106
00007FFFFFFFD800                  [InlinedCallFrame: 00007fffffffd800] NetNativeLibLoader.dll!NetNativeLibLoader.Loader.dl+Unix.dlopen(System.String, NetNativeLibLoader.Loader.SymbolFlag)
00007FFFFFFFD800                  [InlinedCallFrame: 00007fffffffd800] NetNativeLibLoader.dll!NetNativeLibLoader.Loader.dl+Unix.dlopen(System.String, NetNativeLibLoader.Loader.SymbolFlag)
00007FFFFFFFD7F0 00007FFF7D4EE106 System.Diagnostics.Process.dll!ILStubClass.IL_STUB_PInvoke(System.String, NetNativeLibLoader.Loader.SymbolFlag) + 262
00007FFFFFFFD8B0 00007FFF7D4EDFCB NetNativeLibLoader.dll!NetNativeLibLoader.Loader.dl.open(System.String, NetNativeLibLoader.Loader.SymbolFlag, Boolean) + 43
00007FFFFFFFD8D0 00007FFF7D4EDF78 NetNativeLibLoader.dll!NetNativeLibLoader.Loader.UnixPlatformLoader.<.ctor>b__7_1(System.String, NetNativeLibLoader.Loader.SymbolFlag) + 56
00007FFFFFFFD900 00007FFF7D4ED431 NetNativeLibLoader.dll!NetNativeLibLoader.Loader.UnixPlatformLoader.LoadLibrary(System.String, NetNativeLibLoader.Loader.SymbolFlag) + 129
00007FFFFFFFD990 00007FFF7D4ED394 NetNativeLibLoader.dll!NetNativeLibLoader.Loader.UnixPlatformLoader.LoadLibraryInternal(System.String) + 36
00007FFFFFFFD9B0 00007FFF7D4ED348 NetNativeLibLoader.dll!NetNativeLibLoader.Loader.PlatformLoaderBase.LoadLibrary(System.String) + 40
00007FFFFFFFD9D0 00007FFF7D4EBF9D Qml.Net.dll!Qml.Net.Runtimes.RuntimeManager.ConfigureRuntimeDirectory(System.String) + 1501
00007FFFFFFFDA40 00007FFF7D4E9D84 Qml.Net.dll!Qml.Net.Runtimes.RuntimeManager.DiscoverOrDownloadSuitableQtRuntime(RuntimeSearchLocation) + 116
00007FFFFFFFDAF0 00007FFF7D4D61C5 Hammer.dll!Program.Main(System.String[]) + 293 [/home/janvorli/test/Hammer/Program.cs @ 1254]
00007FFFFFFFDC00 00007FFFF69A5447 libcoreclr.so!___lldb_unnamed_symbol9517$$libcoreclr.so + 124
00007FFFFFFFDC20 00007FFFF6806ECB libcoreclr.so!___lldb_unnamed_symbol4410$$libcoreclr.so + 1643
00007FFFFFFFDE00 00007FFFF66D5E3B libcoreclr.so!___lldb_unnamed_symbol300$$libcoreclr.so + 923
00007FFFFFFFE010 00007FFFF66D6169 libcoreclr.so!___lldb_unnamed_symbol301$$libcoreclr.so + 393
00007FFFFFFFE290 00007FFFF67141A3 libcoreclr.so!___lldb_unnamed_symbol1146$$libcoreclr.so + 627
00007FFFFFFFE370 00007FFFF66BEC2D libcoreclr.so!coreclr_execute_assembly + 413
00007FFFFFFFE3E0 00007FFFF7F23821 libhostpolicy.so!___lldb_unnamed_symbol160$$libhostpolicy.so + 1345
00007FFFFFFFE490 00007FFFF7F23CB1 libhostpolicy.so!___lldb_unnamed_symbol161$$libhostpolicy.so + 49
00007FFFFFFFE4C0 00007FFFF7F246DC libhostpolicy.so!corehost_main + 172
00007FFFFFFFE5A0 00007FFFF7F7CED9 libhostfxr.so!___lldb_unnamed_symbol94$$libhostfxr.so + 1817
00007FFFFFFFE6D0 00007FFFF7F7B5DB libhostfxr.so!___lldb_unnamed_symbol92$$libhostfxr.so + 667
00007FFFFFFFE7B0 00007FFFF7F77EBB libhostfxr.so!hostfxr_main_startupinfo + 171
00007FFFFFFFE820 0000555555563B2A dotnet!___lldb_unnamed_symbol148$$dotnet + 938
00007FFFFFFFE8D0 0000555555563FA0 dotnet!___lldb_unnamed_symbol149$$dotnet + 144
00007FFFFFFFE910 00007FFFF6D6B840 libc.so.6!__libc_start_main + 240 at /build/glibc-S7Ft5T/glibc-2.23/csu/libc-start.c:325
00007FFFFFFFE9D0 0000555555558D3A dotnet!___lldb_unnamed_symbol32$$dotnet + 41

janvorli · 2021-06-02T14:13:31Z

It seems that the NetNativeLibLoader needs to be fixed to handle Linux and other Unixes differently and use private const string LibraryNameLinux = "libdl.so.2";

vadi2 · 2021-06-03T05:19:36Z

Thanks so much for the detailed explanation! Sounds like it was a fluke before that it was working with .NET < 5 then?

janvorli · 2021-06-03T08:11:38Z

Is it possible that you were testing the .NET < 5 version installed without snap? That would explain why it worked, as it would use only libraries from the current OS. Actually, you can use a non-snap version of .NET 5 and you would not hit the issue, if it is an option for you.
You can also publish your app as self-contained, in which case the runtime gets packaged with it and again, snap would not cause trouble here.

vadi2 · 2021-06-04T06:49:36Z

Yep - that worked, thank you. I only used snap since it was the simplest to install, but installing the .tar.gz's by hand was not bad at all. I'll submit the loading issue to QML.NET as well.

Really appreciate the in-depth investigation @janvorli, you went above and beyond!

adamsitnik · 2021-07-01T12:19:35Z

Since there is nothing we can do on our end and the issue has been reported in the faulty project (pauldotknopf/net-native-lib-loader#3) I am going to close the issue. Thanks!

dotnet-issue-labeler bot added area-System.IO untriaged New issue has not been triaged by the area owner labels May 26, 2021

vadi2 mentioned this issue Jun 4, 2021

Wrong libdl.so loaded on Linux/Snap pauldotknopf/net-native-lib-loader#3

Open

adamsitnik closed this as completed Jul 1, 2021

adamsitnik added area-AssemblyLoader-coreclr and removed area-System.IO untriaged New issue has not been triaged by the area owner labels Jul 1, 2021

ghost locked as resolved and limited conversation to collaborators Jul 31, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`libdl.so` not found in search path and causes a crash #53291

`libdl.so` not found in search path and causes a crash #53291

vadi2 commented May 26, 2021

ghost commented May 26, 2021

Description

Configuration

Regression?

Other information

janvorli commented Jun 1, 2021

vadi2 commented Jun 1, 2021

janvorli commented Jun 1, 2021

vadi2 commented Jun 1, 2021

janvorli commented Jun 1, 2021 •

edited

Loading

vadi2 commented Jun 2, 2021

janvorli commented Jun 2, 2021

vadi2 commented Jun 2, 2021

janvorli commented Jun 2, 2021

janvorli commented Jun 2, 2021

vadi2 commented Jun 2, 2021

janvorli commented Jun 2, 2021

vadi2 commented Jun 2, 2021

janvorli commented Jun 2, 2021

vadi2 commented Jun 2, 2021

vadi2 commented Jun 2, 2021

janvorli commented Jun 2, 2021

vadi2 commented Jun 2, 2021

janvorli commented Jun 2, 2021

vadi2 commented Jun 2, 2021

janvorli commented Jun 2, 2021

janvorli commented Jun 2, 2021

janvorli commented Jun 2, 2021

vadi2 commented Jun 3, 2021

janvorli commented Jun 3, 2021

vadi2 commented Jun 4, 2021

adamsitnik commented Jul 1, 2021

libdl.so not found in search path and causes a crash #53291

libdl.so not found in search path and causes a crash #53291

Comments

vadi2 commented May 26, 2021

Description

Configuration

Regression?

Other information

ghost commented May 26, 2021

Description

Configuration

Regression?

Other information

janvorli commented Jun 1, 2021

vadi2 commented Jun 1, 2021

janvorli commented Jun 1, 2021

vadi2 commented Jun 1, 2021

janvorli commented Jun 1, 2021 • edited Loading

vadi2 commented Jun 2, 2021

janvorli commented Jun 2, 2021

vadi2 commented Jun 2, 2021

janvorli commented Jun 2, 2021

janvorli commented Jun 2, 2021

vadi2 commented Jun 2, 2021

janvorli commented Jun 2, 2021

vadi2 commented Jun 2, 2021

janvorli commented Jun 2, 2021

vadi2 commented Jun 2, 2021

vadi2 commented Jun 2, 2021

janvorli commented Jun 2, 2021

vadi2 commented Jun 2, 2021

janvorli commented Jun 2, 2021

vadi2 commented Jun 2, 2021

janvorli commented Jun 2, 2021

janvorli commented Jun 2, 2021

janvorli commented Jun 2, 2021

vadi2 commented Jun 3, 2021

janvorli commented Jun 3, 2021

vadi2 commented Jun 4, 2021

adamsitnik commented Jul 1, 2021

`libdl.so` not found in search path and causes a crash #53291

`libdl.so` not found in search path and causes a crash #53291

janvorli commented Jun 1, 2021 •

edited

Loading