[BUG] with ndk r23 and newer, builtin symbols cannot be found when a program is linked with libtool

[Grimler91]

Hi,

Description

Moving from ndk r21 to r23 we started having issues [1], [2], [3], [4], [5], with some programs failing to be run on device with errors like:

CANNOT LINK EXECUTABLE "tesseract": cannot locate symbol "__extenddftf2" referenced by "/data/data/com.termux/files/usr/lib/libtesseract.so"...

In both cases the issue seem to only happen if the code is compiled using a configure && make system, with a CMakeLists.txt it works fine. After some digging we suspect this happens due to libtool and how it handles the linker flags (see discussion).

One (rather ugly) workaround to solve the issue is to add libclang_rt.builtins-aarch64-android.a as a lib in LDFLAGS. Is there a better solution?

Steps to reproduce

The example below is for aarch64.

# Clone a minimal configure&make project that uses `static_cast<long double>` (which compiles into __extenddftf2)
git clone https://github.com/grimler91/minimal-ndk-r23-issue-example 
cd minimal-ndk-r23-issue-example

aclocal
libtoolize --copy
autoconf
automake --add-missing --foreign

# Adjust for path to your ndk
TOOLCHAINROOT=/opt/android-ndk/toolchains/llvm/prebuilt/linux-x86_64
CCPREFIX=$TOOLCHAINROOT/bin/aarch64-linux-android28-

# Optimization level seem to make a difference
CXXFLAGS=-O0 CXX=${CCPREFIX}clang++ CC=${CCPREFIX}clang ./configure --host=aarch64-linux-android --prefix=$(pwd)

make
make install

# Push compiled program, library and libc++_shared to device
adb push bin/main lib/libtest.so $TOOLCHAINROOT/sysroot/usr/lib/aarch64-linux-android/libc++_shared.so /data/local/tmp/

# Try to run the program on device:
adb shell
$ cd /data/local/tmp
$ LD_PRELOAD=./libtest.so:./libc++_shared.so ./main

You should get something like:

WARNING: linker: Warning: "/data/local/tmp/libc++_shared.so" unused DT entry: unknown processor-specific (type 0x70000001 arg 0x0) (ignoring)
CANNOT LINK EXECUTABLE "./main": cannot locate symbol "__extenddftf2" referenced by "/data/local/tmp/libtest.so"...

The __extend* symbols are undefined in libtest.so when compiled like above:

$ readelf -Wa lib/libtest.so | grep __extend
0000000000002938  0000000400000402 R_AARCH64_JUMP_SLOT    0000000000000000 __extenddftf2 + 0
     4: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND __extenddftf2
    35: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND __extenddftf2

but for ndk r21, where we linked against libgcc, we get

$ readelf -Wa lib/libtest.so | grep __extend
0000000000001fd0  0000000e00000402 R_AARCH64_JUMP_SLOT    0000000000000704 __extenddftf2 + 0
    14: 0000000000000704   248 FUNC    GLOBAL DEFAULT   10 __extenddftf2
    73: 0000000000000704   248 FUNC    GLOBAL DEFAULT   10 __extenddftf2

Environment Details

• NDK Version: 23.1.7779620
• Build system: see above
• Host OS: Linux (ubuntu/archlinux)
• ABI: arm64-v8a
• NDK API level: Tested with 24 and 28
• Device API level: 30

[Grimler91]

I've added a minimal CMakeLists.txt as well to the repo. Compiling the program and library with it instead:

mkdir cmake-build && cd cmake-build
CXXFLAGS=-O0 CXX=${CCPREFIX}clang++ CC=${CCPREFIX}clang cmake ..
make

gives a working libtest.so and main executable:

$ readelf -Wa libtest.so | grep __extend
    32: 00000000000016bc   172 FUNC    LOCAL  HIDDEN    12 __extenddftf2

[Grimler91]

Is there any good way to detect this issue without running the program on device?
I could check for UND symbols in every compiled library with something like:

for arch in aarch64 arm i686 x86_64; do
    if [ "$arch" != arm ]; then
        CC=$arch-linux-android-clang
    else
        CC=$arch-linux-androideabi-clang
    fi

    symbols="$(readelf -Wa $($CC -print-libgcc-file-name) | grep "FUNC    GLOBAL HIDDEN" | awk '{print $8}')"
    libraries="$(find path/to/libs/for/$arch -name "*.so")"

    for lib in $libraries; do 
        echo $lib; 
        for sym in $symbols; do 
            readelf -Wa $lib | grep "UND $sym"
        done
    done
done

This seem to work for all arches except arm. On arm we get false positives (in the sense that the program, gettext in this case, works fine even though these looks like undefined symbols):

lib/libasprintf.so
     6: 00000000     0 FUNC    GLOBAL DEFAULT  UND __aeabi_memcpy@LIBC_N (3)
lib/libgettextpo.so
     4: 00000000     0 FUNC    GLOBAL DEFAULT  UND __aeabi_memcpy@LIBC_N (3)
    45: 00000000     0 FUNC    GLOBAL DEFAULT  UND __aeabi_memcpy4@LIBC_N (3)
   102: 00000000     0 FUNC    GLOBAL DEFAULT  UND __aeabi_memcpy8@LIBC_N (3)
    45: 00000000     0 FUNC    GLOBAL DEFAULT  UND __aeabi_memcpy4@LIBC_N (3)
   102: 00000000     0 FUNC    GLOBAL DEFAULT  UND __aeabi_memcpy8@LIBC_N (3)
     5: 00000000     0 FUNC    GLOBAL DEFAULT  UND __aeabi_memmove@LIBC_N (3)
    19: 00000000     0 FUNC    GLOBAL DEFAULT  UND __aeabi_memmove4@LIBC_N (3)
    19: 00000000     0 FUNC    GLOBAL DEFAULT  UND __aeabi_memmove4@LIBC_N (3)
    59: 00000000     0 FUNC    GLOBAL DEFAULT  UND __aeabi_memclr8@LIBC_N (3)
    92: 00000000     0 FUNC    GLOBAL DEFAULT  UND __aeabi_memclr@LIBC_N (3)
    59: 00000000     0 FUNC    GLOBAL DEFAULT  UND __aeabi_memclr8@LIBC_N (3)
    28: 00000000     0 FUNC    GLOBAL DEFAULT  UND __aeabi_memset@LIBC_N (3)
lib/libgettextsrc-0.21.so
     4: 00000000     0 FUNC    GLOBAL DEFAULT  UND __aeabi_memcpy@LIBC_N (3)
    43: 00000000     0 FUNC    GLOBAL DEFAULT  UND __aeabi_memcpy4@LIBC_N (3)
    43: 00000000     0 FUNC    GLOBAL DEFAULT  UND __aeabi_memcpy4@LIBC_N (3)
    65: 00000000     0 FUNC    GLOBAL DEFAULT  UND __aeabi_memmove4@LIBC_N (3)
    65: 00000000     0 FUNC    GLOBAL DEFAULT  UND __aeabi_memmove4@LIBC_N (3)
    84: 00000000     0 FUNC    GLOBAL DEFAULT  UND __aeabi_memset@LIBC_N (3)
lib/libtextstyle.so
lib/libgettextlib.so

is this type of search the best way to detect the issue?

[enh-google]

it's a hack, but you could probably just hard-code the list from https://android.googlesource.com/platform/bionic/+/master/libc/arch-arm/bionic/__aeabi.c and ignore them?

[Grimler91]

@enh-google yes, alright, thanks

[DanAlbert]

@Grimler91 have you had a chance to test this with r24 or r25 yet by any chance? We haven't had a chance to dig into this yet and we're getting r23c ready, so wondering where I should retriage to.

[Grimler91]

@DanAlbert both r24 and r25-beta2 seem to be affected as well (at least for aarch64)

[makotokato]

• If removing -lgcc from linker option on NDK r21, this issue occurs even if NDK r21.
• When removing -nostdlib from linker option on NDK r25, this doesn't seem to occur.

[DanAlbert]

When removing -nostdlib from linker option on NDK r25, this doesn't seem to occur.

Oh, well that option generally shouldn't be used, and in the rare cases where it's necessary, it's up to whoever is passing that option to make sure they get the rest right. If that's the problem here I think we can close this.

[DanAlbert]

Yes, whatever system is forcing -nostdlib into the link command (without correctly reimplementing the driver behaviors that flag disables, but really that's pointless and the flag should just not be used) is the thing with the bug. That doesn't appear anywhere in the source of the repro case, so that's a bug in one of these tools:

aclocal
libtoolize --copy
autoconf
automake --add-missing --foreign