ldd libcoreclr.so failing on Alpine Linux

[ghost]

Among all the .so files produced by CoreCLR, ldd has issue with libcoreclr.so; it complains:

ldd: bin/Product/Linux.x64.Debug/libcoreclr.so: Not a valid dynamic program

It seems like ld.so (the linker) does mprotect(..., rwx) if there's DT_TEXTREL. strace reveals an EACCES when doing mprotect:

alp:~/coreclr$ strace ldd bin/Product/Linux.x64.Debug/libcoreclr.so 
execve("/usr/bin/ldd", ["ldd", "bin/Product/Linux.x64.Debug/libc"...], [/* 16 vars */]) = 0
arch_prctl(ARCH_SET_FS, 0x6b63936e4c28) = 0
set_tid_address(0x6b63936e4c60)         = 14408
open("bin/Product/Linux.x64.Debug/libcoreclr.so", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\374\f\0\0\0\0\0"..., 960) = 960
mmap(NULL, 18423808, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x6b63922c8000
mmap(0x6b63933bc000, 647168, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0xef4000) = 0x6b63933bc000
mmap(0x6b639342e000, 180224, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x6b639342e000
mprotect(0x6b63922c8000, 18423808, PROT_READ|PROT_WRITE|PROT_EXEC) = -1 EACCES (Permission denied)
munmap(0x6b63922c8000, 18423808)        = 0
writev(2, [{"ldd: bin/Product/Linux.x64.Debug"..., 76}, {NULL, 0}], 2ldd: bin/Product/Linux.x64.Debug/libcoreclr.so: Not a valid dynamic program
) = 76
exit_group(1)                           = ?
+++ exited with 1 +++

Checking for text relocations results in:

alp:~/coreclr$ readelf -a bin/Product/Linux.x64.Debug/libcoreclr.so | grep TEXTREL
 0x0000000000000016 (TEXTREL)            0x0

And scalelf -t outputs:

alp:~/coreclr$ scanelf -t bin/Product/Linux.x64.Debug/libcoreclr.so
 TYPE   TEXTREL FILE 
ET_DYN TEXTREL bin/Product/Linux.x64.Debug/libcoreclr.so 

scanelf -qT is even weirder:

alp:~/coreclr$ scanelf -qT bin/Product/Linux.x64.Debug/libcoreclr.so
scanelf: scanelf_file_textrels(): ELF bin/Product/Linux.x64.Debug/libcoreclr.so has TEXTREL markings but doesnt appear to have any real TEXTREL's !?
  bin/Product/Linux.x64.Debug/libcoreclr.so

Here is the full output of readelf -a: https://bpaste.net/raw/c268701a9312 (WARNING: 6.7 MBs of raw text, better use curl -O https://bpaste.net/raw/c268701a9312 and grep the intended section)

To build CoreCLR from scratch on Alpine, https://gist.github.com/jasonwilliams200OK/7d6f5594d3bf697a27c9c1036d349fce.

/cc @amonakov, @richfelker, @barthalion

[ghost]

@jkotas, please let me know if there is more info required to diagnose this issue. :)

[ghost]

Useful link: https://wiki.gentoo.org/wiki/Hardened/Textrels_Guide (note qfile is not available on Alpine, rest of the tools mentioned in the guide are available)

[ghost]

The linker command for libcoreclr.so produced by cmake is:

(executed in dir: coreclr/bin/obj/Linux.x64.Debug/src/dlls/mscoree/coreclr)

/usr/bin/clang++ -fPIC -Wall -Wno-null-conversion -std=c++11 -g -O0 -Wl,--version-script=/home/peterj/coreclr/bin/obj/Linux.x64.Debug/src/dlls/mscoree/coreclr/coreclr.exports -Wl,--build-id=sha1 -Xlinker -Bsymbolic -Xlinker -Bsymbolic-functions -shared -Wl,-soname,libcoreclr.so -o libcoreclr.so CMakeFiles/coreclr.dir/__/mscoree.cpp.o CMakeFiles/coreclr.dir/__/unixinterface.cpp.o CMakeFiles/coreclr.dir/__/__/__/__/version.cpp.o ../../../utilcode/dyncrt/libutilcode.a -Wl,--start-group ../../../debug/ee/wks/libcordbee_wks.a ../../../debug/debug-pal/libdebug-pal.a ../../../unwinder/wks/libunwinder_wks.a ../../../vm/wks/libcee_wks.a ../../../gc/wks/libgc_wks.a -Wl,--end-group ../../../md/compiler/wks/libmdcompiler_wks.a ../../../md/runtime/wks/libmdruntime_wks.a ../../../md/enc/wks/libmdruntimerw_wks.a ../../../md/hotdata/full/libmdhotdata_full.a ../../../classlibnative/bcltype/libbcltype.a ../../../md/ceefilegen/libceefgen.a ../../../classlibnative/float/libcomfloat_wks.a ../../../inc/libcorguids.a ../../../gcinfo/lib/libgcinfo.a ../../../debug/ildbsymlib/libildbsymlib.a ../../../strongname/api/wks/libstrongname_wks.a ../../../utilcode/dyncrt/libutilcode.a ../../../binder/v3binder/libv3binder.a -Wl,--whole-archive ../../../pal/src/libcoreclrpal.a ../../../pal/src/libtracepointprovider.a -Wl,--no-whole-archive ../../mscorrc/full/libmscorrc_debug.a ../../../palrt/libpalrt.a ../../../pal/src/eventprovider/libeventprovider.a ../../../nativeresources/libnativeresourcestring.a -lgcc_s -lpthread -lrt -ldl -luuid -lunwind -lunwind-generic -lunwind-x86_64 -lintl

• If I run this command in coreclr/bin/obj/Linux.x64.Debug/src/dlls/mscoree/coreclr dir as is, ldd libcoreclr.so says "Not a valid dynamic program".
• If I remove these two ../../../vm/wks/libcee_wks.a ../../../gc/wks/libgc_wks.a static libs, ldd works (just complains about some missing symbols).
• If I add ../../../vm/wks/libcee_wks.a only, then it says "Not a valid dynamic program".
• If I add ../../../gc/wks/libgc_wks.a only, then the process hangs forever.

Seems like the issue lie in either ../../../vm/wks/libcee_wks.a or ../../../gc/wks/libgc_wks.a ..

[ghost]

Ah, it was grsec thingy. So I ran: paxctl -c then paxctl -psm for:

• $HOME/core-setup/.dotnet_stage0/Linux/dotnet
• $HOME/core-setup/.dotnet_stage0/Linux/shared/Microsoft.NETCore.App/1.0.0/dotnet
• $HOME/core-setup/.dotnet_stage0/Linux/shared/Microsoft.NETCore.App/1.0.0/corehost
• $HOME/core-setup/.dotnet_stage0/Linux/sdk/1.0.0-preview3-003180/corehost

and it passed that step. Not sure if there is anything we can do here to support hardended/grsec kernels OOTB (without tearing the security protection).

Onto the next step (getting Failed to initialize CoreCLR, HRESULT: 0x80004005, guess this is due to mscorlib.dll and mscorlib.ni.dll which came from Ubuntu tar are mismatching the rest of the build). I'll follow up at #4210.

[richfelker]

This looks like a linker bug. My guess is that there's a relocation that would have been a textrel if -Bsymbolic hadn't bound it at link-time, and the linker wrongly retained the conclusion that textrels were needed even after dropping it.

[ghost]

@richfelker, would be nice if we could get to cc someone here who deals with ld stuff on musl-powered operating systems. I already have my hands muddy with this stuff, can provide more logs/info if need be.:smile:

[richfelker]

Can you first confirm which linker (bfd, gold, or lld) is being used by clang when it links? That will determine where the bug report needs to go. Passing -v on the linking command line is probably sufficient to see that but if not strace would be another method.

[ghost]

@richfelker, with -v to clang++ linker command above, I get:

clang version 3.8.0 (tags/RELEASE_380/final)
Target: x86_64-alpine-linux-musl
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-alpine-linux-musl/5.3.0
Found candidate GCC installation: /usr/lib/gcc/x86_64-alpine-linux-musl/5.3.0
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-alpine-linux-musl/5.3.0
Candidate multilib: .;@m64
Selected multilib: .;@m64
 "/usr/bin/ld" -z now -z relro --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -shared -o libcoreclr.so /usr/bin/../lib/gcc/x86_64-alpine-linux-musl/5.3.0/../../../crti.o /usr/bin/../lib/gcc/x86_64-alpine-linux-musl/5.3.0/crtbeginS.o -L/usr/bin/../lib/gcc/x86_64-alpine-linux-musl/5.3.0 -L/usr/bin/../lib/gcc/x86_64-alpine-linux-musl/5.3.0/../../../../x86_64-alpine-linux-musl/lib -L/usr/bin/../lib/gcc/x86_64-alpine-linux-musl/5.3.0/../../.. -L/usr/bin/../lib -L/lib -L/usr/lib --version-script=/home/peterj/coreclr/bin/obj/Linux.x64.Debug/src/dlls/mscoree/coreclr/coreclr.exports --build-id=sha1 -Bsymbolic -Bsymbolic-functions -soname libcoreclr.so CMakeFiles/coreclr.dir/__/mscoree.cpp.o CMakeFiles/coreclr.dir/__/unixinterface.cpp.o CMakeFiles/coreclr.dir/__/__/__/__/version.cpp.o ../../../utilcode/dyncrt/libutilcode.a --start-group ../../../debug/ee/wks/libcordbee_wks.a ../../../debug/debug-pal/libdebug-pal.a ../../../unwinder/wks/libunwinder_wks.a ../../../vm/wks/libcee_wks.a -Bsymbolic-functions --end-group ../../../md/compiler/wks/libmdcompiler_wks.a ../../../md/runtime/wks/libmdruntime_wks.a ../../../md/enc/wks/libmdruntimerw_wks.a ../../../md/hotdata/full/libmdhotdata_full.a ../../../classlibnative/bcltype/libbcltype.a ../../../md/ceefilegen/libceefgen.a ../../../classlibnative/float/libcomfloat_wks.a ../../../inc/libcorguids.a ../../../gcinfo/lib/libgcinfo.a ../../../debug/ildbsymlib/libildbsymlib.a ../../../strongname/api/wks/libstrongname_wks.a ../../../utilcode/dyncrt/libutilcode.a ../../../binder/v3binder/libv3binder.a --whole-archive ../../../pal/src/libcoreclrpal.a ../../../pal/src/libtracepointprovider.a --no-whole-archive ../../mscorrc/full/libmscorrc_debug.a ../../../palrt/libpalrt.a ../../../pal/src/eventprovider/libeventprovider.a ../../../nativeresources/libnativeresourcestring.a -lgcc_s -lpthread -lrt -ldl -luuid -lunwind -lunwind-generic -lunwind-x86_64 -lintl -lstdc++ -lm -lgcc_s -lc -lgcc_s /usr/bin/../lib/gcc/x86_64-alpine-linux-musl/5.3.0/crtendS.o /usr/bin/../lib/gcc/x86_64-alpine-linux-musl/5.3.0/../../../crtn.o

Here is the strace: https://bpaste.net/show/60e76ae46172

Last but not the least:

$ file /usr/bin/ld
/usr/bin/ld: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, stripped

[ghost]

Version info:

$ ld --version
GNU ld (GNU Binutils) 2.26.20160125
Copyright (C) 2015 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.