Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FR] UBSAN print_stacktrace=1 #954

Open
arturbac opened this issue Apr 7, 2019 · 30 comments
Open

[FR] UBSAN print_stacktrace=1 #954

arturbac opened this issue Apr 7, 2019 · 30 comments
Assignees

Comments

@arturbac
Copy link

arturbac commented Apr 7, 2019

ubsan has a feature originaly to print stack trace of UB.
however on android port this seems to not work print_stacktrace=1 to ASAN and UBSAN but this doesn't seem to produce a stack trace like on linux in logcat

I added to wrap.sh

export UBSAN_OPTIONS=print_stacktrace=1

This sometimes is required to fix a bug for example when UB is detected in libc++ but source of ub happens much earlier in code calls.
Example of useles information without stack trace.

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /opt/android-ndk-r20/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/c++/v1/algorithm:1636:21

NDK tested r20 beta 1 platform i686

@enh
Copy link
Contributor

enh commented Apr 8, 2019

@eugenis

@eugenis
Copy link

eugenis commented Apr 8, 2019

Could you reproduce it with a standalone binary? Works fine for me with ndk r19.

@DanAlbert
Copy link
Member

Is the problem maybe that the trace only goes to stderr and not logcat?

@arturbac
Copy link
Author

arturbac commented Apr 9, 2019

exactly @DanAlbert logcat contains only SUMARY and last line of stack where the error was finaly went into action.
Does stderr can be intercepted for ordinary java with native native android app ? how ?

@DanAlbert
Copy link
Member

What device? Probably more interesting, what OS version? I could have sworn we switched the common sanitizer logging over to syslog ages ago (L or M). syslog as of about that time should print to logcat.

@arturbac
Copy link
Author

arturbac commented Apr 9, 2019

I tried asan/ubsan on
xiaomi redmi 4x + lineageos 15.1 (8.1 android) - no luck , it fails with null ptr in loadlibrary

it works for me only on emulator i686 with android 9.0 goggle apis and i686 build

i tried also emualtor x86_64 with 64 build but it fails to run with Shadow memory range interleaves with an existing memory mapping. ASan cannot proceed correctly. ABORTING. As i read it is known error and i cannot do anything , just have to wait for new kernel ..

@arturbac
Copy link
Author

arturbac commented Apr 9, 2019

My wrap.sh (i build with cmake/gradle)
app/src/main/resources/x86/wrap.sh

HERE="$(cd "$(dirname "$0")" && pwd)"
export ASAN_OPTIONS=log_to_syslog=false,allow_user_segv_handler=1,detect_stack_use_after_return=1,print_stacktrace=1
export UBSAN_OPTIONS=print_stacktrace=1
export LD_PRELOAD="$HERE/libclang_rt.asan-i686-android.so"
"$@"

@arturbac
Copy link
Author

arturbac commented Apr 9, 2019

@DanAlbert You mentioned "syslog as of about that time should print to logcat."
But example wrap.sh in ndk contains line log_to_syslog=false, is it correct ?

cat /opt/android-ndk-r20/wrap.sh/asan.x86.sh

HERE="$(cd "$(dirname "$0")" && pwd)"
export ASAN_OPTIONS=log_to_syslog=false,allow_user_segv_handler=1
export LD_PRELOAD=$HERE/libclang_rt.asan-i686-android.so
"$@"

@arturbac
Copy link
Author

arturbac commented Apr 9, 2019

Could you reproduce it with a standalone binary? Works fine for me with ndk r19.

I use ASAN & UBSAN only for java android app with native library

@enh
Copy link
Contributor

enh commented Apr 10, 2019

the log_to_syslog=false is in the canonical docs too: https://github.com/google/sanitizers/wiki/AddressSanitizerOnAndroid

@eugenis --- do you remember the reason for this? should we just delete this from both places, since it seems like it's not useful for the typical [app] developer. (i guess that option only really makes sense if you're running a gtest directly?)

@eugenis
Copy link

eugenis commented Apr 11, 2019 via email

@arturbac
Copy link
Author

arturbac commented Apr 11, 2019

Maybe the problem with logcat stack trace is that it is to long to push it at once, and get croped to single line.
When i put custom stacktrace with boost::stacktrace to logcat I need to iterate line by line when pushing to logcat.

sout << " " << boost::stacktrace::stacktrace();
for (std::string line; std::getline(input, line); )
  LOG("%s", line);

adb logcat -g
main: ring buffer is 4Mb (3Mb consumed), max entry is 5120b, max payload is 4068b
system: ring buffer is 4Mb (0b consumed), max entry is 5120b, max payload is 4068b
crash: ring buffer is 4Mb (15Kb consumed), max entry is 5120b, max payload is 4068b

@DanAlbert
Copy link
Member

I didn't realize that logwrapper was used by default with wrap.sh, but the code agrees with you.

I can't speak to why this doesn't work on LineageOS, but given that it does work on Android I think this is something you'll need to take up with LineageOS.

@arturbac
Copy link
Author

arturbac commented Apr 11, 2019

No w8.
UBSAN doesn't produce full satcktrace on emulator i686, it produces only 1 stack trace line as i show.
ASAN doesn't work at all on emulator x86_64, but stick to i686, i just need at least one device like emulator 686 to produce USBAN errors with stack traces.

@DanAlbert DanAlbert reopened this Apr 11, 2019
@arturbac
Copy link
Author

I think i know why it doesn't work.
The method described placing wrap.sh with gradle/AS in resources/lib.ARCH_name
at https://developer.android.com/ndk/guides/wrap-script
simply doesn't work at all on device and emulator, the wrap.sh placed in apk
is placed in folder name as folder name in resources, not in lib/ARCH-name/ but side to it in lib.ARCH_name.
When I did asan_device_setup everything works as expected on x86 and x86_64 emulators.
But I had to fix outdated asan_device_script

  • add missing x86_64
  • rename arch i386 to i686
  • extend internal script creation and add UBSAN_OPTIONS

Concluding
whats the proper way of placing wrap.sh with gradle/android studio ?

@enh
Copy link
Contributor

enh commented Apr 15, 2019

what version of Studio are you using?

@enh
Copy link
Contributor

enh commented Apr 15, 2019

the Studio folks tell me i got the docs wrong... apparently although it will show up as "libs.x86" under resources, it should actually be resources/libs/x86 on disk.

here's my suggested corrected text for the docs:

Android Studio only packages `.so` files from the `lib/` directories, so if
you're an Android Studio user, you'll need to place your `wrap.sh` files in the
`src/main/resources/lib/*` directories instead, so that they'll be packaged
correctly. Note that `resources/lib/x86` will be displayed in the UI as
`lib.x86`, but it should actually be a subdirectory:
[[existing image]]

can you confirm that if you create libs/x86 rather than libs.x86 (etc) that it works?

@arturbac
Copy link
Author

arturbac commented Apr 15, 2019

Thats was the point on filesystem this must be located
app/src/main/resources/lib/ARCH/wrap.sh
then wrap.sh in apk is packaged with output native libs
and I can confirm that i see UBSAN stacktraces bit different from asan_prepare_device but it looks liek it works ok.

the differences in logcat
asan prepared device:

  • taged with application tag x.y.z
  • no escape color codes

wrap.sh

  • taged with wrap.sh tag instead of app tag
  • color escape codes

Do You maybe know how to disable escape color codes ?
I dont see any option here
https://github.com/google/sanitizers/wiki/AddressSanitizerFlags

04-15 21:47:24.798  4463  4463 I wrap.sh :   0xe6d20690: �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m
04-15 21:47:24.798  4463  4463 I wrap.sh :   0xe6d206a0: �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m
04-15 21:47:24.798  4463  4463 I wrap.sh :   0xe6d206b0: �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[35mfd�[1m�[0m
04-15 21:47:24.798  4463  4463 I wrap.sh :   0xe6d206c0: �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[35mfd�[1m�[0m
04-15 21:47:24.798  4463  4463 I wrap.sh :   0xe6d206d0: �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[0m04�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[35mfd�[1m�[0m
04-15 21:47:24.798  4463  4463 I wrap.sh : =>0xe6d206e0: �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[0m04�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m[�[1m�[0m03�[1m�[0m]�[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[35mfd�[1m�[0m
04-15 21:47:24.798  4463  4463 I wrap.sh :   0xe6d206f0: �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[35mfd�[1m�[0m
04-15 21:47:24.798  4463  4463 I wrap.sh :   0xe6d20700: �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[0m00�[1m�[0m �[1m�[0m00�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[35mfd�[1m�[0m
04-15 21:47:24.798  4463  4463 I wrap.sh :   0xe6d20710: �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m
04-15 21:47:24.798  4463  4463 I wrap.sh :   0xe6d20720: �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[35mfd�[1m�[0m
04-15 21:47:24.798  4463  4463 I wrap.sh :   0xe6d20730: �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[31mfa�[1m�[0m �[1m�[35mfd�[1m�[0m �[1m�[35mfd�[1m�[0m
04-15 21:47:24.798  4463  4463 I wrap.sh : Shadow byte legend (one shadow byte represents 8 application bytes):

@enh
Copy link
Contributor

enh commented Apr 15, 2019

does

LD_PRELOAD="$HERE/libclang_rt.asan-i686-android.so" exec "$@"

instead of

LD_PRELOAD="$HERE/libclang_rt.asan-i686-android.so" "$@"

fix the tag?

@eugenis should be able to answer the question about the escape sequences...

@eugenis
Copy link

eugenis commented Apr 15, 2019 via email

@arturbac
Copy link
Author

arturbac commented Apr 15, 2019

the exec doesn't change tag from wrap.sh however its not a problem for me.

To be more precise for futre readers the wrap.sh must look like below with exec in new line

LD_PRELOAD="$HERE/libclang_rt.asan-i686-android.so"
exec "$@"

Thank You very much @enh @eugenis @DanAlbert for help.

My final wrap.sh for future readers

#!/system/bin/sh
HERE="$(cd "$(dirname "$0")" && pwd)"
export ASAN_OPTIONS=log_to_syslog=false,allow_user_segv_handler=1,detect_stack_use_after_return=1,check_initialization_order=true,quarantine_size_mb=64,color=never
export UBSAN_OPTIONS=print_stacktrace=1,log_to_syslog=false,color=never
export LD_PRELOAD="$HERE/libclang_rt.asan-i686-android.so"
exec "$@"

@arturbac
Copy link
Author

arturbac commented Apr 17, 2019

From my point of view we can close this FR
Conclusion to source problem
For gradle/cmake projects the wrap.sh must must be located on disk at
app/src/main/resources/lib/ABI/wrap.sh
where ABI means one of x86,x86_64,arm64-v8a depending for which ABi is script (LD_PRELOAD library)

@DanAlbert
Copy link
Member

I'll leave it open so we remember to fix the docs

@DanAlbert DanAlbert reopened this Apr 17, 2019
@arturbac
Copy link
Author

and toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/8.0.7/bin/asan_device_setup

  • fixi386 ->i686
  • add missing x86_64

@DanAlbert
Copy link
Member

I'm not sure there's much value in keeping asan_device_setup in a world where we have wrap.sh.

@arturbac
Copy link
Author

arturbac commented Apr 17, 2019

From a an app developer who prepares tools for team.
asan_device_setup has and advantage that sanitized run in emualtor is easier to prepare, doesn't need any changes to gradle config and build.
only simple cmake parameter of configuration for -fsanitize flags.

wrap.sh scripts are problematic as they cannot be a static resources.
depending on build type i must dynamicaly change source tree and modify resources during build.
I hope maybe someday gradle/jave will alllow out of source building. :-)
with cmake + asan_device_setup , there is no source tree change and asan_device_setup needs oly once to be run for emulator.

wrap.sh wouldn't be so problematic if wrap.sh could be palced inside direcotory of native libs , but with gradle it cannot, gradle ignores everything except.so

I'm afraid (paranoic) with wrap.sh and dynamic soruce tree modification from gradle cofnig that with some error in gradle config of app public release build will contain wrap.sh when no one will remember in team about wrap.sh and do some change ..
Of course there is layer of testers but I like to avoid errors by design and prefer dyamic code in build dir instead of soruce tree modification.

@DanAlbert
Copy link
Member

Fair enough. I suppose at least until gradle makes this less of a PITA there's value in keeping it around.

@eugenis: do you want to make the asan_device_setup fixes, or should I send you a patch?

@eugenis
Copy link

eugenis commented Apr 19, 2019 via email

@DanAlbert
Copy link
Member

The asan_device_setup patch that I never got around to sending should be less helpful now that hwasan is the thing that people should be using, right? Should I close this?

@enh-google
Copy link
Collaborator

The asan_device_setup patch that I never got around to sending should be less helpful now that hwasan is the thing that people should be using, right? Should I close this?

seems like the original submitter was using x86 and x86-64? no hwasan for them :-(

(but this is a pretty niche script that most people can't run, and we haven't heard anyone else notice it was broken, so probably not worth much effort.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants