[BUG] [NDK R25c] [Android-13] multi-thread with asan fatal error : signal 4 (SIGILL), code 2 (ILL_ILLOPN)

[DaydreamCoding]

Description

use asan in Android-13 with multi-thread fatal error

NDK R22b pass
NDK R25c error

   : tagged_addr_ctrl: 0000000000000001 (PR_TAGGED_ADDR_ENABLE)
   : pac_enabled_keys: 000000000000000f (PR_PAC_APIAKEY, PR_PAC_APIBKEY, PR_PAC_APDAKEY, PR_PAC_APDBKEY)
   : signal 4 (SIGILL), code 2 (ILL_ILLOPN), fault addr 0x000000710635bf50
   :     x0  0000000000000000  x1  0000000000000001  x2  0000000000000000  x3  0000000000000000
   :     x4  0000000000000000  x5  0000000000000000  x6  0000007075f98d8a  x7  7f7f7f7f7f7f7f7f
   :     x8  00000000000000a7  x9  0000000000000010  x10 0000000000000000  x11 000000710a48c386
   :     x12 0000000000000034  x13 0000007075f98a85  x14 0000000000000000  x15 000000000000000a
   :     x16 000000710a573930  x17 000000710635be6c  x18 00000070706e2000  x19 0000007075f98cb0
   :     x20 000000710a78e000  x21 0000007075f98cb0  x22 000000000000561a  x23 000000000000561a
   :     x24 0000007075f98cb0  x25 0000007075f98cb0  x26 0000007075f98ff8  x27 00000000000fc000
   :     x28 00000000000fe000  x29 0000007075f98c50
   :     lr  003f90710a561544  sp  0000007075f98c50  pc  000000710635bf50  pst 0000000020001000
   : backtrace:
   :       #00 pc 0000000000079f50  /data/local/tmp/libclang_rt.asan-aarch64-android.so (__interceptor_prctl+228) (BuildId: d39265be9efeee419e565f66dd253078e6302c36)
   :       #01 pc 00000000000f8cac  [anon:stack_and_tls:22043]

Affected versions

r25

Canary version

No response

Host OS

Mac

Host OS version

macOS 13.2.1

Affected ABIs

arm64-v8a

Build system

CMake

Other build system

No response

minSdkVersion

33

Device API level

No response

[enh-google]

that's a PAC failure, no?

i think you have some generated code here:

   :       #01 pc 00000000000f8cac  [anon:stack_and_tls:22043]

that's incorrect?

[DanAlbert]

If it turns out to be something else, we will need a repro case.

[DaydreamCoding]

device list : only some device cause error

• xiaomi 13 T (android 13, api 33), soc : SM7450
• xiaomi 13 (android 13, api 33), soc : SM8550
• xiaomi 12 (android 12, api 31), soc : SM8450
• vivo x90 pro (android 13, api 33), soc : MT6985

repo :

• git clone -b feature/ndk_r25_asan https://github.com/DaydreamCoding/neon-intrinsics-test.git

steps:

export ANDROID_NKD=`your r25c ndk path`
./build/android_build_Release.sh  # build
./build/run_android64.sh  # will push NDK R25 libclang_rt.asan-aarch64-android.so to device

when remove this thread error gone : https://github.com/DaydreamCoding/neon-intrinsics-test/blob/feature/ndk_r25_asan/test/image-warpTest.cpp#L15

--------- beginning of crash
10-19 03:55:10.270  5819  5820 F libc    : Fatal signal 4 (SIGILL), code 2 (ILL_ILLOPN), fault addr 0x70cecb4f50 in tid 5820 (test), pid 5819 (test)
10-19 03:55:10.292  5824  5824 I crash_dump64: obtaining output fd from tombstoned, type: kDebuggerdTombstoneProto
10-19 03:55:10.294  1209  1209 I tombstoned: received crash request for pid 5820
10-19 03:55:10.295  5824  5824 I crash_dump64: performing dump of process 5819 (target tid = 5820)
10-19 03:55:10.315  5824  5824 F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
10-19 03:55:10.315  5824  5824 F DEBUG   : Build fingerprint: 'Xiaomi/fuxi/fuxi:13/TKQ1.220905.001/V14.0.0.22.10.17.TMCCNXM:user/release-keys'
10-19 03:55:10.277   949   949 I logd    : logdr: UID=0 GID=0 PID=5824 n tail=0 logMask=8 pid=5819 start=0ns deadline=0ns
10-19 03:55:10.278   949   949 I logd    : logdr: UID=0 GID=0 PID=5824 n tail=0 logMask=1 pid=5819 start=0ns deadline=0ns
10-19 03:55:10.315  5824  5824 F DEBUG   : Revision: '0'
10-19 03:55:10.315  5824  5824 F DEBUG   : ABI: 'arm64'
10-19 03:55:10.315  5824  5824 F DEBUG   : Timestamp: 2022-10-19 03:55:10.297651372+0800
10-19 03:55:10.315  5824  5824 F DEBUG   : Process uptime: 1s
10-19 03:55:10.315  5824  5824 F DEBUG   : Cmdline: /data/local/tmp/sample/arm64-v8a/test
10-19 03:55:10.315  5824  5824 F DEBUG   : pid: 5819, tid: 5820, name: test  >>> /data/local/tmp/sample/arm64-v8a/test <<<
10-19 03:55:10.315  5824  5824 F DEBUG   : uid: 0
10-19 03:55:10.315  5824  5824 F DEBUG   : tagged_addr_ctrl: 0000000000000001 (PR_TAGGED_ADDR_ENABLE)
10-19 03:55:10.315  5824  5824 F DEBUG   : pac_enabled_keys: 000000000000000f (PR_PAC_APIAKEY, PR_PAC_APIBKEY, PR_PAC_APDAKEY, PR_PAC_APDBKEY)
10-19 03:55:10.316  5824  5824 F DEBUG   : signal 4 (SIGILL), code 2 (ILL_ILLOPN), fault addr 0x00000070cecb4f50
10-19 03:55:10.316  5824  5824 F DEBUG   :     x0  0000000000000000  x1  0000000000000001  x2  0000000000000000  x3  0000000000000000
10-19 03:55:10.316  5824  5824 F DEBUG   :     x4  0000000000000000  x5  0000000000000000  x6  000000704d7a8dda  x7  7f7f7f7f7f7f7f7f
10-19 03:55:10.316  5824  5824 F DEBUG   :     x8  00000000000000a7  x9  0000000000000010  x10 0000000000000000  x11 00000070d2a43386
10-19 03:55:10.316  5824  5824 F DEBUG   :     x12 0000000000000032  x13 000000704d7a8ad4  x14 0000000000000000  x15 000000000000000a
10-19 03:55:10.316  5824  5824 F DEBUG   :     x16 00000070d2b2a930  x17 00000070cecb4e6c  x18 000000704c324000  x19 000000704d7a8d00
10-19 03:55:10.316  5824  5824 F DEBUG   :     x20 00000070d2d45000  x21 000000704d7a8d00  x22 00000000000016bb  x23 00000000000016bb
10-19 03:55:10.316  5824  5824 F DEBUG   :     x24 000000704d7a8d00  x25 000000704d7a8d00  x26 000000704d7a9008  x27 00000000000fb000
10-19 03:55:10.316  5824  5824 F DEBUG   :     x28 00000000000fd000  x29 000000704d7a8ca0
10-19 03:55:10.316  5824  5824 F DEBUG   :     lr  004d0470d2b18544  sp  000000704d7a8ca0  pc  00000070cecb4f50  pst 0000000020001000
10-19 03:55:10.316  5824  5824 F DEBUG   : backtrace:
10-19 03:55:10.316  5824  5824 F DEBUG   :       #00 pc 0000000000079f50  /data/local/tmp/sample/arm64-v8a/libclang_rt.asan-aarch64-android.so (__interceptor_prctl+228) (BuildId: d39265be9efeee419e565f66dd253078e6302c36)
10-19 03:55:10.316  5824  5824 F DEBUG   :       #01 pc 00000000000f7cfc  [anon:stack_and_tls:5820]

@enh-google @DanAlbert

If it turns out to be something else, we will need a repro case.

[DanAlbert]

What about #1848 (comment)?

[DaydreamCoding]

device list : only some device cause error

• xiaomi 13 T (android 13, api 33), soc : SM7450
• xiaomi 13 (android 13, api 33), soc : SM8550
• xiaomi 12 (android 12, api 31), soc : SM8450
• vivo x90 pro (android 13, api 33), soc : MT6985

all devices are ARMv9

[DanAlbert]

Yes, only armv9 supports PAC, and this looks like a PAC failure in generated code in your app.

afaict the only docs that cover this are https://developer.android.com/ndk/guides/abis#armv9_enabling_pac_and_bti_for_cc. I've filed a bug (http://b/272807546, couldn't find a public component for it, sorry) for documenting that better.

[eugenis]

ILL_ILLOPN on arm64 is always PAC. Not sure why the top frame is in ASan?

[enh-google]

Yes, only armv9 supports PAC, and this looks like a PAC failure in generated code in your app.

...and before you argue "no, PAC is an armv8.3 feature!", yes, that's strictly true, but it wasn't implemented in any of Arm's core before the armv9 cores (except for Apple's iPhone SoCs, which don't use Arm's cores).

ILL_ILLOPN on arm64 is always PAC. Not sure why the top frame is in ASan?

i think because they're saying it only reproduces with asan?

here's the code they link to:

    auto fun = []() {
        printf("test thread\n");
        usleep(0.5 * 1000 * 1000);
    };
    auto thread = std::thread(fun);
    thread.join();

[DaydreamCoding]

Yes, only armv9 supports PAC, and this looks like a PAC failure in generated code in your app.

only failure with NDK R25 asan library (libclang_rt.asan-aarch64-android.so), use system library or others are normaly run

afaict the only docs that cover this are https://developer.android.com/ndk/guides/abis#armv9_enabling_pac_and_bti_for_cc. I've filed a bug (http://b/272807546, couldn't find a public component for it, sorry) for documenting that better.

if build with asan, not found PAC by llvm-readelf --notes

if without asan and with -mbranch-protection=standard normaly run, found PAC by llvm-readelf --notes

if with asan and with -mbranch-protection=standard, not found PAC by llvm-readelf --notes

here's the code they link to:

    auto fun = []() {
        printf("test thread\n");
        usleep(0.5 * 1000 * 1000);
    };
    auto thread = std::thread(fun);
    thread.join();

yes, only use thread + asan library (NDK R25) failure, use system library or others are normaly run