Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CoreCLR native build on Linux / Linux_musl arm64 / arm causes clang crashes #84802

Closed
trylek opened this issue Apr 13, 2023 · 8 comments
Closed
Labels
area-Infrastructure-coreclr blocking-outerloop Blocking the 'runtime-coreclr outerloop' and 'runtime-libraries-coreclr outerloop' runs

Comments

@trylek
Copy link
Member

trylek commented Apr 13, 2023

OS: Linux / Linux_musl
Architecture: arm64, arm

Example run:
https://dev.azure.com/dnceng-public/public/_build/results?buildId=237422&view=logs&j=fd9dce65-fed6-5f8b-f099-1d407f24e80f&t=73b092a7-fc6e-5522-101b-81c9e52247b8

Going over the recent runs it seems to be pretty recent - first repro I've found is from yesterday afternoon:
https://dev.azure.com/dnceng-public/public/_build/results?buildId=237112&view=results

Diagnostics: clang crashes when compiling src/native/external/libunwind/src/elf32.c or elf64.c based on targeting architecture. I don't see any recent changes to their source code so I tend to suspect either a clang version update or some scripting update. The problem may not be fully deterministic - the above example run is the one where this also happens on Linux arm, in the remaining runs I saw it was only reproing on Linux / Linux_musl arm64.

PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.	Program arguments: /usr/local/bin/clang-12 --target=arm-linux-gnueabihf --gcc-toolchain=/crossrootfs/arm/usr --sysroot=/crossrootfs/arm -DBUILDENV_DEBUG=1 -DCLR_ARM_FPU_CAPABILITY=0x7 -DDEBUG -DDISABLE_CONTRACTS -DHAVE_CONFIG_H=1 -DHAVE___THREAD=0 -DHOST_ARM -DHOST_UNIX -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_STRING=\"\" -DTARGET_32BIT -DTARGET_ARM -DTARGET_LINUX -DTARGET_UNIX -DURTBLDENV_FRIENDLY=Debug -DUSE_STL -D_DBG -D_DEBUG -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -Darm_search_unwind_table=UNW_OBJ(arm_search_unwind_table) -Dasm=__asm__ -I/__w/1/s/artifacts/obj/external/libunwind -I/__w/1/s/src/native/external/libunwind_extras -I/__w/1/s/src/native -I/__w/1/s/src/coreclr/pal/inc -I/__w/1/s/src/coreclr/pal/src -I/__w/1/s/src/coreclr/pal/../inc -I/__w/1/s/src/native/external/libunwind/include -I/__w/1/s/src/native/external/libunwind/include/tdep -I/__w/1/s/artifacts/obj/external/libunwind/include -I/__w/1/s/artifacts/obj/external/libunwind/include/tdep -I/__w/1/s/src/native/external/libunwind/include/tdep-arm -I/__w/1/s/src/native/external/libunwind/src -g -fPIC -mthumb -mfpu=vfpv3 -O0 -g -Wall -Wno-null-conversion -fno-omit-frame-pointer -fms-extensions -fwrapv -fstack-protector-strong -Werror -Wno-unused-variable -Wno-unused-value -Wno-unused-function -Wno-tautological-compare -Wno-unknown-pragmas -Wimplicit-fallthrough -Wno-unused-but-set-variable -ffp-contract=off -Wno-unknown-warning-option -ferror-limit=4096 -Wno-unused-private-field -Wno-constant-logical-operand -Wno-pragma-pack -Wno-incompatible-ms-struct -Wno-reserved-identifier -Wno-unsafe-buffer-usage -Wno-single-bit-bitfield-constant-conversion -Wno-cast-function-type-strict -fsigned-char -fvisibility=hidden -ffunction-sections -march=armv7-a -fexceptions -Wno-format -Wno-format-security -Wno-implicit-fallthrough -Wno-header-guard -Wno-absolute-value -Wno-inline-asm -Wno-implicit-function-declaration -Wno-implicit-int-conversion -std=gnu11 -MD -MT /__w/1/s/artifacts/obj/external/libunwind/CMakeFiles/libunwind.dir/__w/1/s/src/native/external/libunwind/src/elf32.c.o -MF CMakeFiles/libunwind.dir/__w/1/s/src/native/external/libunwind/src/elf32.c.o.d -o CMakeFiles/libunwind.dir/__w/1/s/src/native/external/libunwind/src/elf32.c.o -c /__w/1/s/src/native/external/libunwind/src/elf32.c
1.	 parser at end of file
2.	Code generation
[  8%] Building CXX object md/enc/CMakeFiles/mdruntimerw_wks_obj.dir/cmake_pch.hxx.pch
 #0 0x000056228ba9805e llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/usr/local/bin/clang-12+0x213f05e)
 #1 0x000056228ba95e54 llvm::sys::RunSignalHandlers() (/usr/local/bin/clang-12+0x213ce54)
 #2 0x000056228ba0c438 CrashRecoverySignalHandler(int) (/usr/local/bin/clang-12+0x20b3438)
 #3 0x00007f29380e9e30 __restore_rt (/lib/libc.so.6+0x42e30)
 #4 0x000056228c53e310 llvm::DIE::getUnitDie() const (/usr/local/bin/clang-12+0x2be5310)
 #5 0x000056228c54869c llvm::DwarfDebug::finishEntityDefinitions() (/usr/local/bin/clang-12+0x2bef69c)
 #6 0x000056228c561a52 llvm::DwarfDebug::finalizeModuleInfo() (/usr/local/bin/clang-12+0x2c08a52)
 #7 0x000056228c564a68 llvm::DwarfDebug::endModule() (/usr/local/bin/clang-12+0x2c0ba68)
 #8 0x000056228c535aa9 llvm::AsmPrinter::doFinalization(llvm::Module&) (/usr/local/bin/clang-12+0x2bdcaa9)
 #9 0x000056228b3ef55d llvm::FPPassManager::doFinalization(llvm::Module&) (.localalias) (/usr/local/bin/clang-12+0x1a9655d)
#10 0x000056228b3fb020 llvm::legacy::PassManagerImpl::run(llvm::Module&) (/usr/local/bin/clang-12+0x1aa2020)
#11 0x000056228bd2bf2c (anonymous namespace)::EmitAssemblyHelper::EmitAssembly(clang::BackendAction, std::unique_ptr >) (.constprop.0) (/usr/local/bin/clang-12+0x23d2f2c)
#12 0x000056228bd2dc7c clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::DataLayout const&, llvm::Module*, clang::BackendAction, std::unique_ptr >) (/usr/local/bin/clang-12+0x23d4c7c)
#13 0x000056228c96670c clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) (/usr/local/bin/clang-12+0x300d70c)
#14 0x000056228d430b39 clang::ParseAST(clang::Sema&, bool, bool) (/usr/local/bin/clang-12+0x3ad7b39)
#15 0x000056228c33b269 clang::FrontendAction::Execute() (/usr/local/bin/clang-12+0x29e2269)
#16 0x000056228c2deccb clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/usr/local/bin/clang-12+0x2985ccb)
#17 0x000056228c3e9c58 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/usr/local/bin/clang-12+0x2a90c58)
#18 0x000056228a50788f cc1_main(llvm::ArrayRef, char const*, void*) (/usr/local/bin/clang-12+0xbae88f)
#19 0x000056228a5052a0 ExecuteCC1Tool(llvm::SmallVectorImpl&) (/usr/local/bin/clang-12+0xbac2a0)
#20 0x000056228c18d5b5 void llvm::function_ref::callback_fn >, std::__cxx11::basic_string, std::allocator >*, bool*) const::'lambda'()>(long) (/usr/local/bin/clang-12+0x28345b5)
#21 0x000056228ba0c533 llvm::CrashRecoveryContext::RunSafely(llvm::function_ref) (/usr/local/bin/clang-12+0x20b3533)
#22 0x000056228c18e6df clang::driver::CC1Command::Execute(llvm::ArrayRef >, std::__cxx11::basic_string, std::allocator >*, bool*) const (.part.0) (/usr/local/bin/clang-12+0x28356df)
#23 0x000056228c164ad7 clang::driver::Compilation::ExecuteCommand(clang::driver::Command const&, clang::driver::Command const*&) const (/usr/local/bin/clang-12+0x280bad7)
#24 0x000056228c165519 clang::driver::Compilation::ExecuteJobs(clang::driver::JobList const&, llvm::SmallVectorImpl >&) const (/usr/local/bin/clang-12+0x280c519)
#25 0x000056228c172f29 clang::driver::Driver::ExecuteCompilation(clang::driver::Compilation&, llvm::SmallVectorImpl >&) (/usr/local/bin/clang-12+0x2819f29)
#26 0x000056228a47ecb2 main (/usr/local/bin/clang-12+0xb25cb2)
#27 0x00007f29380d557d __libc_start_call_main (/lib/libc.so.6+0x2e57d)
#28 0x00007f29380d5630 __libc_start_main_impl (/lib/libc.so.6+0x2e630)
#29 0x000056228a504c85 _start (/usr/local/bin/clang-12+0xbabc85)
clang-12: error: clang frontend command failed with exit code 139 (use -v to see invocation)
clang version 12.0.1
Target: arm-unknown-linux-gnueabihf
Thread model: posix
InstalledDir: /usr/local/bin
[  8%] Building CXX object md/compiler/CMakeFiles/mdcompiler_wks_obj.dir/assemblymd.cpp.o
clang-12: note: diagnostic msg: 
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang-12: note: diagnostic msg: /tmp/elf32-2e3faa.c
clang-12: note: diagnostic msg: /tmp/elf32-2e3faa.sh
clang-12: note: diagnostic msg: 

********************

I must admit I don't yet fully understand what the "parser at end of file" refers to, whether it's the compiler command line or the source file.

/cc @dotnet/runtime-infrastructure @janvorli

@trylek trylek added area-Infrastructure-coreclr blocking-outerloop Blocking the 'runtime-coreclr outerloop' and 'runtime-libraries-coreclr outerloop' runs labels Apr 13, 2023
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Apr 13, 2023
@ghost
Copy link

ghost commented Apr 13, 2023

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

OS: Linux / Linux_musl
Architecture: arm64, arm

Example run:
https://dev.azure.com/dnceng-public/public/_build/results?buildId=237422&view=logs&j=fd9dce65-fed6-5f8b-f099-1d407f24e80f&t=73b092a7-fc6e-5522-101b-81c9e52247b8

Going over the recent runs it seems to be pretty recent - first repro I've found is from yesterday afternoon:
https://dev.azure.com/dnceng-public/public/_build/results?buildId=237112&view=results

Diagnostics: clang crashes when compiling src/native/external/libunwind/src/elf32.c or elf64.c based on targeting architecture. I don't see any recent changes to their source code so I tend to suspect either a clang version update or some scripting update. The problem may not be fully deterministic - the above example run is the one where this also happens on Linux arm, in the remaining runs I saw it was only reproing on Linux / Linux_musl arm64.

PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.	Program arguments: /usr/local/bin/clang-12 --target=arm-linux-gnueabihf --gcc-toolchain=/crossrootfs/arm/usr --sysroot=/crossrootfs/arm -DBUILDENV_DEBUG=1 -DCLR_ARM_FPU_CAPABILITY=0x7 -DDEBUG -DDISABLE_CONTRACTS -DHAVE_CONFIG_H=1 -DHAVE___THREAD=0 -DHOST_ARM -DHOST_UNIX -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_STRING=\"\" -DTARGET_32BIT -DTARGET_ARM -DTARGET_LINUX -DTARGET_UNIX -DURTBLDENV_FRIENDLY=Debug -DUSE_STL -D_DBG -D_DEBUG -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -Darm_search_unwind_table=UNW_OBJ(arm_search_unwind_table) -Dasm=__asm__ -I/__w/1/s/artifacts/obj/external/libunwind -I/__w/1/s/src/native/external/libunwind_extras -I/__w/1/s/src/native -I/__w/1/s/src/coreclr/pal/inc -I/__w/1/s/src/coreclr/pal/src -I/__w/1/s/src/coreclr/pal/../inc -I/__w/1/s/src/native/external/libunwind/include -I/__w/1/s/src/native/external/libunwind/include/tdep -I/__w/1/s/artifacts/obj/external/libunwind/include -I/__w/1/s/artifacts/obj/external/libunwind/include/tdep -I/__w/1/s/src/native/external/libunwind/include/tdep-arm -I/__w/1/s/src/native/external/libunwind/src -g -fPIC -mthumb -mfpu=vfpv3 -O0 -g -Wall -Wno-null-conversion -fno-omit-frame-pointer -fms-extensions -fwrapv -fstack-protector-strong -Werror -Wno-unused-variable -Wno-unused-value -Wno-unused-function -Wno-tautological-compare -Wno-unknown-pragmas -Wimplicit-fallthrough -Wno-unused-but-set-variable -ffp-contract=off -Wno-unknown-warning-option -ferror-limit=4096 -Wno-unused-private-field -Wno-constant-logical-operand -Wno-pragma-pack -Wno-incompatible-ms-struct -Wno-reserved-identifier -Wno-unsafe-buffer-usage -Wno-single-bit-bitfield-constant-conversion -Wno-cast-function-type-strict -fsigned-char -fvisibility=hidden -ffunction-sections -march=armv7-a -fexceptions -Wno-format -Wno-format-security -Wno-implicit-fallthrough -Wno-header-guard -Wno-absolute-value -Wno-inline-asm -Wno-implicit-function-declaration -Wno-implicit-int-conversion -std=gnu11 -MD -MT /__w/1/s/artifacts/obj/external/libunwind/CMakeFiles/libunwind.dir/__w/1/s/src/native/external/libunwind/src/elf32.c.o -MF CMakeFiles/libunwind.dir/__w/1/s/src/native/external/libunwind/src/elf32.c.o.d -o CMakeFiles/libunwind.dir/__w/1/s/src/native/external/libunwind/src/elf32.c.o -c /__w/1/s/src/native/external/libunwind/src/elf32.c
1.	 parser at end of file
2.	Code generation
[  8%] Building CXX object md/enc/CMakeFiles/mdruntimerw_wks_obj.dir/cmake_pch.hxx.pch
 #0 0x000056228ba9805e llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/usr/local/bin/clang-12+0x213f05e)
 #1 0x000056228ba95e54 llvm::sys::RunSignalHandlers() (/usr/local/bin/clang-12+0x213ce54)
 #2 0x000056228ba0c438 CrashRecoverySignalHandler(int) (/usr/local/bin/clang-12+0x20b3438)
 #3 0x00007f29380e9e30 __restore_rt (/lib/libc.so.6+0x42e30)
 #4 0x000056228c53e310 llvm::DIE::getUnitDie() const (/usr/local/bin/clang-12+0x2be5310)
 #5 0x000056228c54869c llvm::DwarfDebug::finishEntityDefinitions() (/usr/local/bin/clang-12+0x2bef69c)
 #6 0x000056228c561a52 llvm::DwarfDebug::finalizeModuleInfo() (/usr/local/bin/clang-12+0x2c08a52)
 #7 0x000056228c564a68 llvm::DwarfDebug::endModule() (/usr/local/bin/clang-12+0x2c0ba68)
 #8 0x000056228c535aa9 llvm::AsmPrinter::doFinalization(llvm::Module&) (/usr/local/bin/clang-12+0x2bdcaa9)
 #9 0x000056228b3ef55d llvm::FPPassManager::doFinalization(llvm::Module&) (.localalias) (/usr/local/bin/clang-12+0x1a9655d)
#10 0x000056228b3fb020 llvm::legacy::PassManagerImpl::run(llvm::Module&) (/usr/local/bin/clang-12+0x1aa2020)
#11 0x000056228bd2bf2c (anonymous namespace)::EmitAssemblyHelper::EmitAssembly(clang::BackendAction, std::unique_ptr >) (.constprop.0) (/usr/local/bin/clang-12+0x23d2f2c)
#12 0x000056228bd2dc7c clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::DataLayout const&, llvm::Module*, clang::BackendAction, std::unique_ptr >) (/usr/local/bin/clang-12+0x23d4c7c)
#13 0x000056228c96670c clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) (/usr/local/bin/clang-12+0x300d70c)
#14 0x000056228d430b39 clang::ParseAST(clang::Sema&, bool, bool) (/usr/local/bin/clang-12+0x3ad7b39)
#15 0x000056228c33b269 clang::FrontendAction::Execute() (/usr/local/bin/clang-12+0x29e2269)
#16 0x000056228c2deccb clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/usr/local/bin/clang-12+0x2985ccb)
#17 0x000056228c3e9c58 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/usr/local/bin/clang-12+0x2a90c58)
#18 0x000056228a50788f cc1_main(llvm::ArrayRef, char const*, void*) (/usr/local/bin/clang-12+0xbae88f)
#19 0x000056228a5052a0 ExecuteCC1Tool(llvm::SmallVectorImpl&) (/usr/local/bin/clang-12+0xbac2a0)
#20 0x000056228c18d5b5 void llvm::function_ref::callback_fn >, std::__cxx11::basic_string, std::allocator >*, bool*) const::'lambda'()>(long) (/usr/local/bin/clang-12+0x28345b5)
#21 0x000056228ba0c533 llvm::CrashRecoveryContext::RunSafely(llvm::function_ref) (/usr/local/bin/clang-12+0x20b3533)
#22 0x000056228c18e6df clang::driver::CC1Command::Execute(llvm::ArrayRef >, std::__cxx11::basic_string, std::allocator >*, bool*) const (.part.0) (/usr/local/bin/clang-12+0x28356df)
#23 0x000056228c164ad7 clang::driver::Compilation::ExecuteCommand(clang::driver::Command const&, clang::driver::Command const*&) const (/usr/local/bin/clang-12+0x280bad7)
#24 0x000056228c165519 clang::driver::Compilation::ExecuteJobs(clang::driver::JobList const&, llvm::SmallVectorImpl >&) const (/usr/local/bin/clang-12+0x280c519)
#25 0x000056228c172f29 clang::driver::Driver::ExecuteCompilation(clang::driver::Compilation&, llvm::SmallVectorImpl >&) (/usr/local/bin/clang-12+0x2819f29)
#26 0x000056228a47ecb2 main (/usr/local/bin/clang-12+0xb25cb2)
#27 0x00007f29380d557d __libc_start_call_main (/lib/libc.so.6+0x2e57d)
#28 0x00007f29380d5630 __libc_start_main_impl (/lib/libc.so.6+0x2e630)
#29 0x000056228a504c85 _start (/usr/local/bin/clang-12+0xbabc85)
clang-12: error: clang frontend command failed with exit code 139 (use -v to see invocation)
clang version 12.0.1
Target: arm-unknown-linux-gnueabihf
Thread model: posix
InstalledDir: /usr/local/bin
[  8%] Building CXX object md/compiler/CMakeFiles/mdcompiler_wks_obj.dir/assemblymd.cpp.o
clang-12: note: diagnostic msg: 
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang-12: note: diagnostic msg: /tmp/elf32-2e3faa.c
clang-12: note: diagnostic msg: /tmp/elf32-2e3faa.sh
clang-12: note: diagnostic msg: 

********************

I must admit I don't yet fully understand what the "parser at end of file" refers to, whether it's the compiler command line or the source file.

/cc @dotnet/runtime-infrastructure @janvorli

Author: trylek
Assignees: -
Labels:

area-Infrastructure-coreclr, blocking-outerloop

Milestone: -

@sbomer
Copy link
Member

sbomer commented Apr 13, 2023

This looks like #84503. I was seeing it in linux arm64 mono build, but hadn't seen it in other builds. dotnet/dotnet-buildtools-prereqs-docker#842 seemed to fix it for me. But the images in dotnet-buildtools-prereqs-docker have already been updated to llvm 16, which we are not yet ready to use. I think we'll want to roll back the images to llvm 12 to fix this ASAP.

@janvorli
Copy link
Member

@sbomer I am a bit confused - the failure logged above was caused by clang 12, why do we need to revert the upgrade to clang 16?

@sbomer
Copy link
Member

sbomer commented Apr 13, 2023

Runtime is currently building with a set of mariner images that have clang 12, built with gcc. There is no published set of images that has clang 12, built with clang - because the llvm 16 upgrade went in before the build-with-clang fix. So I'm reverting the upgrade temporarily to get us a new set of images that have clang-built clang 12.

@janvorli
Copy link
Member

Ah, got it, thank you for the explanation.

@trylek
Copy link
Member Author

trylek commented Apr 13, 2023

Frankly speaking I'm not getting why it doesn't fail the same way in the runtime (PR / CI) pipeline, e.g. in this Egor's PR run started about half an hour ago:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=238687&view=results

@sbomer
Copy link
Member

sbomer commented Apr 13, 2023

I'm not sure either - I was able to reproduce it consistently when I saw the failure building mono arm64. But it might be very sensitive to the inputs.

@sbomer
Copy link
Member

sbomer commented Apr 14, 2023

I believe this is fixed with #84851. Please reopen if you hit this again.

@sbomer sbomer closed this as completed Apr 14, 2023
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Apr 14, 2023
@ghost ghost locked as resolved and limited conversation to collaborators May 15, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-Infrastructure-coreclr blocking-outerloop Blocking the 'runtime-coreclr outerloop' and 'runtime-libraries-coreclr outerloop' runs
Projects
None yet
Development

No branches or pull requests

3 participants