stdlib: Missing backtrace on panic when using std::backtrace::Backtrace vs backtrace::Backtrace on Android

[sgasse]

When compiling standalone binaries for Android with debug information, there seems to be no backtrace information available. However it is possible to get a backtrace when using backtrace as external dependency.

I am building native binaries for Android devices. I noticed that I could not get a backtrace on panic. This is reproducible for me on a physical Samsung Galaxy Tab A9 (aarch64-linux-android, with Android 13 and Android API 33), on a Pixel 3a emulator (x86_64-linux-android with Android 14 and Android API 34) as well as on another target device for which I cannot disclose version information.

For the rest of the report, I will only provide the instructions for the Samsung Galaxy Tab A9, but the same behavior happened with the emulator and the other Android device.

Meta

$ rustc --version --verbose
rustc 1.75.0 (82e1608df 2023-12-21)
binary: rustc
commit-hash: 82e1608dfa6e0b5569232559e3d385fea5a93112
commit-date: 2023-12-21
host: x86_64-unknown-linux-gnu
release: 1.75.0
LLVM version: 17.0.6

Setup

Here is a sample of code for a test binary called stack_trace:

fn outer_func0() {
    inner_func(false);
}

fn outer_func1() {
    inner_func(true);
}

fn inner_func(param: bool) {
    if param {
        panic!("Intentional panic");
    }
}

fn main() {
    outer_func0();
    outer_func1();
}

For linking, I tried two versions.

• Using clang as part of the latest Android NDK.
• Using the linker coming with cross (apparently -C linker=aarch64-linux-android-gcc)

# .cargo/config.toml
# For the Galaxy Tab A9
[target.aarch64-linux-android]
linker = "/home/user/android-ndk-r26c/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android33-clang"

Building

# With plan cargo
cargo build --target aarch64-linux-android

# With cross
cross build --target aarch64-linux-android

Issue

When running RUST_BACKTRACE=1 ./stack_trace, I would expect to see the backtrace of the panic, telling me which call chain led to the panic.

Instead, I see this output (backtrace is missing):

$ RUST_BACKTRACE=1 ./stack_trace                                               
thread 'main' panicked at src/main.rs:11:9:
Intentional panic
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Analysis

You may wonder if the debug symbols got stripped, but they were not:

$ file stack_trace                                                         
stack_trace: ELF shared object, 64-bit LSB arm64, dynamic (/system/bin/linker64), for Android 33, built by NDK r26c (11394342), not stripped

So after some searching, I stumbled over this comment by @zarik5 so I tried using backtrace directly, setting a custom panic hook:

fn panic_with_backtrace_rs() {
    std::panic::set_hook(Box::new(|panic_info| {
        print!(
            "thread '{}' panicked",
            std::thread::current().name().unwrap_or("unknown")
        );

        if let Some(location) = panic_info.location() {
            println!(" at {}:{}:", location.file(), location.line());
        } else {
            println!("");
        }

        if let Some(payload) = panic_info.payload().downcast_ref::<&str>() {
            println!("{}", payload);
        }

        if let Ok("1") = std::env::var("RUST_BACKTRACE").as_deref() {
            println!("{:?}", backtrace::Backtrace::new());
        }
    }));
}

If I call this function in main and build with cargo (linking with clang), I get this output:

$ RUST_BACKTRACE=1 ./stack_trace                                           
thread 'main' panicked at src/main.rs:11:
Intentional panic
   0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: <unknown>
  13: <unknown>
  14: <unknown>
  15: <unknown>

And if I keep the custom hook and build with cross (linking with gcc), I finally get a usable backtrace:

$ RUST_BACKTRACE=1 ./stack_trace                                           
thread 'main' panicked at src/main.rs:11:
Intentional panic
   0: stack_trace::panic_with_backtrace_rs::{{closure}}
             at /home/user/workspace/examples/stack_trace/src/main.rs:40:30
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/alloc/src/boxed.rs:2021:9
      std::panicking::rust_panic_with_hook
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:783:13
   2: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:649:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:170:18
   4: rust_begin_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:645:5
   5: core::panicking::panic_fmt
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panicking.rs:72:14
   6: stack_trace::inner_func
             at /home/user/workspace/examples/stack_trace/src/main.rs:11:9
   7: stack_trace::outer_func1
             at /home/user/workspace/examples/stack_trace/src/main.rs:6:5
   8: stack_trace::main
             at /home/user/workspace/examples/stack_trace/src/main.rs:19:5
   9: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
  10: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
  11: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
  12: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
      std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
      std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
      std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
      std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
      std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
      std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  13: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  14: main
  15: __libc_init

Now you may assume it has something to do with cross. But on the third Android device which I tested (for which I cannot disclose further info), we also use clang for linking with a specific NDK version. There, I also get the "unknown" stack trace when using the default panic hook and a proper one when using backtrace as external dependency - however I do not need to compile with cross, it also works with cargo.

So unfortunately, I need to include cross in the example here to reproduce it, but I have reason to assume that this is not related to cross and the issue happens the same when linking with clang. Without a physical device, I was also able to reproduce it in the Android emulator (came as part of Android Studio). Given that I am apparently not the first person missing backtraces on Android I wonder, is a bug in the stdlib's usage of backtrace for Android?

[jieyouxu]

@rustbot label +O-android +E-needs-mcve -needs-triage

[fengys1996]

I also encountered the same problem.

After some investigation, I guess:

backtrace-rs is introduced into the std lib in this way, which will cause the build.rs of backtrace-rs to not be executed, and the dl_iterate_phdr feature will not be set. Ultimately, the stack information cannot be obtained.

[rib]

Ah, nice, good find @fengys1996

I can confirm that I can also get a working backtrace on Android if I:

1. explicitly install a panic handler and manually use backtrace-rs as described above
2. make sure to enable legacy jni packaging / android:extractNativeLibs="true"
3. ensure debug symbols aren't stripped

In my case I'm not using cross or cargo ndk but we have our own cargo build tool / wrapper that makes sure to set up the environment variables that the cc crate expects.

If the issue is related to the build.rs code linked above then I think it makes sense @sgasse that you won't get a valid backtrace with cargo build --target aarch64-linux-android since that's not setting up CC/CXX environment variables for the cc crate which is used by the backtrace-rs build.rs script.

@sgasse you could potentially try explicitly depending on the backtrace crate with features = [ "dl_iterate_phdr" ] if trying to use vanilla cargo build --target aarch64-linux-android or manually setting up the environment variables for the cc crate.

I think for dl_iterate_phdr to be usable on Android then jni libraries need to be unpacked from the .apk when installed instead of being directly mapped from .apk archive, which is a deprecated packaging option that needs to be explicitly enabled.

While I'm using Gradle to build Android packages, with different buildTypes, then I added something like this to test:

        debug {
            packagingOptions {
                jniLibs {
                    useLegacyPackaging = true
                }
                doNotStrip '**/*.so'
            }
            debuggable true
        }

I think it could also work to set android:extractNativeLibs="true" in your AndroidManifest.xml (for the <application )