[BUG] Cannot resolve symbols from binary linked with LLD #1196

mikaheli · 2020-02-28T09:55:06Z

Description

Cannot resolve symbols (crash location) from crash dump from binary that is linked with -fuse=lld. I'm using ndk-stack to resolve symbols from crash dump. When using current default linker from ndk, things work fine.

Here is small example to reproduce:

Structure:

├── application
│   ├── application.cpp
│   └── CMakeLists.txt
└── build_and_run.sh

application/application.cpp

#include <cstdlib>

int main(void)
{
	abort();
}

application/CMakeLists.txt

cmake_minimum_required(VERSION 3.10)

project(application)
add_executable(application application.cpp)

project(application-lld)
add_executable(application-lld application.cpp)
target_link_libraries(application-lld
	PUBLIC
		-fuse-ld=lld
)

build_and_run.sh

#!/usr/bin/env bash

set -e

HERE=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )

ANDROID_NDK=$HOME/Android/Sdk/ndk/21.0.6113669
ANDROID_ABI=arm64-v8a
ANDROID_NATIVE_API_LEVEL=29

function compile()
{
	printf '\nConfigure & Compile test applications ...\n\n'

	mkdir "$HERE/build"

	cd "$HERE/build"

	cmake \
		-G Ninja \
		-DCMAKE_BUILD_TYPE:STRING=Debug \
		-DBUILD_SHARED_LIBS:BOOL=ON \
		-DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake \
		-DANDROID_ABI=$ANDROID_ABI \
		-DANDROID_NATIVE_API_LEVEL=$ANDROID_NATIVE_API_LEVEL \
		-DANDROID_TOOLCHAIN=clang \
		-DANDROID_STL=c++_static \
		-DCMAKE_RUNTIME_OUTPUT_DIRECTORY=$HERE/bin \
		"$HERE/application"

	cmake --build . -- -j1 -v

	printf '\nConfigure & Compile test applications DONE\n\n'
}

function run()
{
	printf '\nRun with & without LLD\n'

	adb logcat -c

	adb push $HERE/bin/application /data/local/tmp/
	adb push $HERE/bin/application-lld /data/local/tmp/

	set +e

	adb shell /data/local/tmp/application
	adb shell /data/local/tmp/application-lld

	set -e
}

function crash_dump()
{
	printf '\nCrash dumps:\n\n'
	adb logcat -d | ndk-stack -sym $HERE/bin
}

compile
run
crash_dump

Output crash dumps:

Default linker

********** Crash dump: **********
Build fingerprint: 'OPPO/PCCM00/OP46C3:9/PKQ1.190223.001/1553783366:user/release-keys'
#00 0x00000000000223bc /system/lib64/libc.so (abort+116)
#01 0x0000000000000704 /data/local/tmp/application
main
/home/mikaheli/Desktop/LLDTEST/application/application.cpp:5:2
#02 0x00000000000ca024 /system/lib64/libc.so (__libc_init+88)
Crash dump is completed

LLD linker

********** Crash dump: **********
Build fingerprint: 'OPPO/PCCM00/OP46C3:9/PKQ1.190223.001/1553783366:user/release-keys'
#00 0x00000000000223bc /system/lib64/libc.so (abort+116)
#01 0x0000000000000094 /data/local/tmp/application-lld (offset 0x1000)
                                                        __libc_init
                                                        ??:0:0
Crash dump is completed

Environment Details

NDK Version: 21.0.6113669
Build system: CMake
Host OS: MX Linux 18.3 / Windows 10
ABI: arm64-v8a / armeabi-v7a
NDK API level: 24/29 tested
Device API level:

The text was updated successfully, but these errors were encountered:

DanAlbert · 2020-04-14T20:28:48Z

@pirama-arumuga-nainar any ideas? I haven't had a chance to test with the latest LLD yet.

HaiPhan2002 · 2020-06-10T09:43:08Z

Hello,

I have the same issue on armeabi-v7a. However, I don't get this issue on arm64-v8a.

Environment Details

NDK Version: 21.0.6113669
Build system: CMake
Host OS:  Windows 10 64bit
ABI: armeabi-v7a
NDK API level: 19/28 tested
Device API level: 21

I made the sample in case you need to recheck it. Link is here

DanAlbert · 2020-06-10T10:02:43Z

Thanks. I'll check that against a more up to date toolchain tomorrow. You could try the canary if you want, but I think the master branch is fairly behind what we'll actually be shipping in r22.

DanAlbert · 2020-06-10T10:04:30Z

Although you could try downloading the LLD from https://android.googlesource.com/platform/prebuilts/clang/host/windows-x86/+/refs/heads/master/clang-r383902/ and replacing the LLD that's in your NDK with that. I think r22 will have something slightly newer, but that's what I'll test tomorrow since I don't have anything newer yet :)

DanAlbert · 2020-06-11T20:33:34Z

It looks like the toolchain we have is going to need further updates, so waiting for the new new one before trying the above.

DanAlbert · 2020-06-19T20:36:37Z

Ended up cherry-picking to the toolchain rather than a full update, and this is still broken with that version:

********** Crash dump: **********
Build fingerprint: 'google/marlin/marlin:7.1.2/NJH47B/4021576:user/release-keys'
#00 0x000000000006b2ec /system/lib64/libc.so (tgkill+8)
#01 0x0000000000068788 /system/lib64/libc.so (pthread_kill+64)
#02 0x0000000000023ce8 /system/lib64/libc.so (raise+24)
#03 0x000000000001c76c /system/lib64/libc.so (abort+52)
#04 0x00000000000006bc /data/local/tmp/application
main
/usr/local/google/home/danalbert/src/ndk/ndk/foo/application/application.cpp:5:2
#05 0x000000000001a594 /system/lib64/libc.so (__libc_init+88)
#06 0x0000000000000674 /data/local/tmp/application
_start_main
??:0:0
Crash dump is completed

********** Crash dump: **********
Build fingerprint: 'google/marlin/marlin:7.1.2/NJH47B/4021576:user/release-keys'
#00 0x000000000006b2ec /system/lib64/libc.so (tgkill+8)
#01 0x0000000000068788 /system/lib64/libc.so (pthread_kill+64)
#02 0x0000000000023ce8 /system/lib64/libc.so (raise+24)
#03 0x000000000001c76c /system/lib64/libc.so (abort+52)
#04 0x0000000000000660 /data/local/tmp/application-lld
??
??:0:0
Crash dump is completed

Going to leave this triaged here for the moment. Could be that we need another toolchain update, could be that this is broken upstream, could be a bug in ndk-stack. We need to know more before we can actually ship lld as the default, I think.

android/ndk#1196

DanAlbert · 2020-06-19T20:56:41Z

https://github.com/DanAlbert/lld-stack-dump-bug is the repro case from the original post, updated slightly to accommodate the changes I'm making to master today and to set the linker from the command line rather than CMakeLists.txt in case the issue was the toolchain file not knowing (it wasn't, but had to check).

DanAlbert · 2020-06-25T21:46:57Z

Seems this is http://b/109657296 (thanks a bunch to @pirama-arumuga-nainar for all the help debugging!). If I add -Wl,--no-rosegment to the link flags it symoblizes fine. The problem is actually the unwinder older devices not understanding the output from LLD.

The fix for the driver that was needed for this never got sent, so we'll patch the build systems and document the problem for now and get the fix in for r23.

DanAlbert · 2020-06-25T22:57:20Z

https://android-review.googlesource.com/c/platform/ndk/+/1350672 is the fix for r22. Still need to write the driver patch.

@mikaheli @Seal2002 you should be able to fix this locally for old NDK versions as well. Just make sure -Wl,--no-rosegment gets added to your linker flags.

DanAlbert · 2020-06-25T23:31:13Z

Okay, we can't actually lift this into the driver until the old linkers are gone. The driver doesn't know what linker its using and neither gold nor bfd accept that argument. LLD doesn't know what version of Android it's targeting, so we can't make the fix there either.

This is just going to be a build system fix until we've removed the old linkers. Filed #1294 to track that.

@jomof fyi you'll want to make this fix in VS as well. See the updated doc in the patch in the previous comment for more info.

Issue: android/ndk#1196 This is a port of the fix from NDK master: https://android-review.googlesource.com/c/platform/ndk/+/1350672 Quoting description from the original fix: > The unwinder used for crash handling on Android devices prior to API 29 cannot > correctly unwind binaries built with `-Wl,--rosegment`. This flag is enabled by > default when using LLD, so if using LLD and targeting devices older than API 29 > you must pass `-Wl,--no-rosegment` when linking for correct stack traces in logcat. ref:3e66526bf1f6bd39b04f067de173c9ca090e5f13

mikaheli added the bug label Feb 28, 2020

DanAlbert added this to the r22 milestone Apr 10, 2020

DanAlbert assigned pirama-arumuga-nainar Apr 14, 2020

DanAlbert added a commit to DanAlbert/lld-stack-dump-bug that referenced this issue Jun 19, 2020

Test case for LLD stack dump issue.

2bcc2e2

android/ndk#1196

DanAlbert closed this as completed Jun 25, 2020

mastakrishna mentioned this issue Jan 11, 2021

[BUG] --no-rosegment not supported by ld.gold from linux64 NDK r22 #1426

Closed

DanAlbert mentioned this issue Sep 27, 2021

[BUG] Broken crash stack traces with NDK r22 and SDK 29 #1589

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Cannot resolve symbols from binary linked with LLD #1196

[BUG] Cannot resolve symbols from binary linked with LLD #1196

mikaheli commented Feb 28, 2020

DanAlbert commented Apr 14, 2020

HaiPhan2002 commented Jun 10, 2020 •

edited

Loading

DanAlbert commented Jun 10, 2020

DanAlbert commented Jun 10, 2020

DanAlbert commented Jun 11, 2020

DanAlbert commented Jun 19, 2020

DanAlbert commented Jun 19, 2020

DanAlbert commented Jun 25, 2020

DanAlbert commented Jun 25, 2020

DanAlbert commented Jun 25, 2020

[BUG] Cannot resolve symbols from binary linked with LLD #1196

[BUG] Cannot resolve symbols from binary linked with LLD #1196

Comments

mikaheli commented Feb 28, 2020

Description

Environment Details

DanAlbert commented Apr 14, 2020

HaiPhan2002 commented Jun 10, 2020 • edited Loading

DanAlbert commented Jun 10, 2020

DanAlbert commented Jun 10, 2020

DanAlbert commented Jun 11, 2020

DanAlbert commented Jun 19, 2020

DanAlbert commented Jun 19, 2020

DanAlbert commented Jun 25, 2020

DanAlbert commented Jun 25, 2020

DanAlbert commented Jun 25, 2020

HaiPhan2002 commented Jun 10, 2020 •

edited

Loading