Crash in libjpeg.so when BitmapFactory.decodeStream called #1058

rauk123 · 2019-08-06T12:23:04Z

Since we need to support 64bit CPU in order to publish on Google Play.Then we found a strange crash happened in libjpeg.so in some devices(arm64-v8a).

Some logs are below:

1 #00 pc 0000000000004a00 /system/lib64/libjpeg.so [arm64-v8a]
2 #01 pc 00000000000189d4 /system/lib64/libjpeg.so [arm64-v8a]
3 #02 pc 000000000001fef4 /system/lib64/libjpeg.so [arm64-v8a]
4 #03 pc 0000000000017188 /system/lib64/libjpeg.so (jpeg_read_scanlines+152) [arm64-v8a]
5 #04 pc 00000000002484e4 /system/lib64/libskia.so (SkJPEGImageDecoder::onDecode(SkStream*, SkBitmap*, SkImageDecoder::Mode)+4532) [arm64-v8a]
6 #05 pc 000000000023d9b4 /system/lib64/libskia.so (SkImageDecoder::decode(SkStream*, SkBitmap*, SkColorType, SkImageDecoder::Mode)+152) [arm64-v8a]
7 #06 pc 00000000000c7b48 /system/lib64/libandroid_runtime.so (android::NativeInputEventSender::NativeInputEventSender(_JNIEnv*, _jobject*, android::sp<android::InputChannel> const&, android::sp<android::MessageQueue> const&)+136) [arm64-v8a]
8 #07 pc 00000000000c857c /system/lib64/libandroid_runtime.so (android::NativeInputEventSender::receiveFinishedSignals(_JNIEnv*)+48) [arm64-v8a]
9 #08 pc 0000000000f647b8 /system/framework/arm64/boot.oat (oatdata+16136120) [arm64-v8a]
10 java:
11 android.graphics.BitmapFactory.decodeStreamInternal(BitmapFactory.java:677)
12 android.graphics.BitmapFactory.decodeStream(BitmapFactory.java:653)
13 com.c.a.utils.be.a(Encore:820)

Any ideas? Thanks a lot.

The text was updated successfully, but these errors were encountered:

alexcohn · 2019-08-07T20:08:54Z

Does your app include libjpeg? Is it your obfuscated Java code com.c.a.utils.be.a that calls BitmapFactory.decodeStream ?

rauk123 · 2019-08-08T02:53:17Z

Does your app include libjpeg? Is it your obfuscated Java code com.c.a.utils.be.a that calls BitmapFactory.decodeStream ?

com.c.a.utils.be.a is the code calles BitmapFactory.decodeStream. App doesn't include libjpeg.so. But some android system uses this libjpeg.so to decode bitmap. And some uses libqc-opt.so. I also found some crashes about libqc-opt.so.

1 #00 pc 0000000000004b6c /system/vendor/lib64/libqc-opt.so [arm64-v8a]
2 #01 pc 00000000000187e0 /system/vendor/lib64/libqc-opt.so [arm64-v8a]
3 #02 pc 000000000001fe30 /system/vendor/lib64/libqc-opt.so [arm64-v8a]
4 #03 pc 0000000000017008 /system/vendor/lib64/libqc-opt.so (jReadScanlines+152) [arm64-v8a]
5 #04 pc 000000000037712c /system/lib64/libskia.so (SkJPEGImageDecoder::onDecode(SkStream*, SkBitmap*, SkImageDecoder::Mode)+1084) [arm64-v8a]
6 #05 pc 000000000023ac48 /system/lib64/libskia.so (SkImageDecoder::decode(SkStream*, SkBitmap*, SkColorType, SkImageDecoder::Mode)+80) [arm64-v8a]
7 #06 pc 00000000000c59f0 /system/lib64/libandroid_runtime.so (android::android_view_InputDevice_create(_JNIEnv*, android::InputDeviceInfo const&)+568) [arm64-v8a]
8 #07 pc 00000000000c613c /system/lib64/libandroid_runtime.so [arm64-v8a]
9 #08 pc 0000000002827288 /data/dalvik-cache/arm64/system@framework@boot.oat (oatexec+6062728) [arm64-v8a]
10 java:
11 android.graphics.BitmapFactory.decodeStream(BitmapFactory.java:630)
12 android.graphics.BitmapFactory.decodeResourceStream(BitmapFactory.java:456)
13 android.graphics.BitmapFactory.decodeResource(BitmapFactory.java:479)
14 com.c.a.utils.be.a(Encore:825)

Unfortunately，we can't reproduce this problem. It was happened on some devices.

alexcohn · 2019-08-08T07:36:33Z

You did not answer the specific question: Does this stack start with your obfuscated Java code? Does your app contain some C++ components, too?

It could be very helpful for the community if you share the names of offending devices. There may be a bug in some vendor-specific implementations of 64-bit libjpeg.so. Note that on the qualcomm device, this libqc-opt.so is simply a wrapper for the same libjpeg.

If the crash reproduce on the same device often, you can try to install a 32-bit version of your app on such device. Will the crashes still happen?

rauk123 · 2019-08-08T10:08:36Z

Thanks.

Yes，this stack starts with my obfuscated Java code. And my app also contains some C++ components.

Before we supported 64-bit version, we never received this bug report. But now, we have to support 64-bit version in order to publish on Google Play. After that, many reports about this bug appeared. So I am sure that it will not crash with a 32-bit version app.

It happened at most on these devices.

alexcohn · 2019-08-08T10:27:55Z

Auch, this hurts. Don't hesitate to write to OPPO and to HTC, let them fix their bugs. I would also suggest you to escalate this issue with your local Google representative. With the new 64-bit regulations, it will definitely hit not only you.

As for immediate practical steps… You use split build, don't you? Then you can set (in build.gradle) minSdkVersion for the 64-bit flavor to 26 or even higher. This will exclude the older devices where libjpeg or other 64-bit system libraries may still have glitches.

rauk123 · 2019-08-08T13:00:29Z

Yes, I agree that it looks like a bug in some OPPO devices.
Thanks for your suggestions. I'll try

srikanthsunkari · 2019-08-09T05:36:25Z

Hi @alexcohn I am facing a similar issue but I observed Oppo Realme 3Pro (RMX1851), Android 9 was the one device which is from many other brands, Mi A1 (tissot_sprout), Android 9 is the most effective device. It could be specific to Android 9, or latest ndk lib since it is only happening on Android 9 version.
And also It doesn't come from our code.

alexcohn · 2019-08-11T09:58:53Z

@srikanthsunkari generally speaking, you can blacklist specific devices for a version on Play Store. Unfortunately, this may become hard to manage if you want to keep the 32-bit flavor of your app available for users with RMX1851 and other problematic devices.

srikanthsunkari · 2019-08-12T10:48:12Z

@alexcohn thank you for your reply, May I know what is RMC1851?

alexcohn · 2019-08-12T12:24:23Z

On Mon, Aug 12, 2019 at 1:48 PM srikanthsunkari ***@***.***> wrote: @alexcohn <https://github.com/alexcohn> thank you for your reply, May I know what is RMC1851?

It's my typo. I wanted to write RMX1851.

…

srikanthsunkari · 2019-08-12T12:43:08Z

@alexcohn , One of our lib uses ndk, But this issue is not from that lib or from our stack trace It is completely from ndk lib.
Can you please provide an explanation for us why this scenario is producible?

alexcohn · 2019-08-12T15:35:31Z

@srikanthsunkari as I wrote above, this seems to be a bug in OPPO system: the way the built the 64-bit libjpeg.so (a system library) has some fault.

DanAlbert · 2019-08-12T17:45:03Z

Thanks a bunch for the help as always, @alexcohn

I sent this thread along to the PM tracking 64-bit transition issues. Given that this appears to be a device-specific issue, I'm not sure there's much we can do on our end short of documenting the advice @alexcohn gave in #1058 (comment)

DanAlbert · 2019-08-12T19:52:28Z

set (in build.gradle) minSdkVersion for the 64-bit flavor to 26 or even higher.

@alexcohn: Where'd you get 26 from? Is that a guess based on those devices all being 25 or lower (I haven't checked all of them), or is there some indication that this was fixed in that release?

We clearly have a missing CTS test somewhere, I'm just not sure what that test is without knowing how to repro.

alexcohn · 2019-08-12T22:07:00Z

@DanAlbert the devices listed in the original table are all 25 and lower. Then came @srikanthsunkari with Realme 3 Pro from the same manufacturer, but with API 28, so my suggestion happens to have been too optimistic.

This is really a 64-bit transition issue. Note that it may be extremely hard for indie developers to handle such rare crashes on a rather exotic phone. I hope that with Google's resources, you can find what really happens in 64-bit Jpeg decoder on these devices. There is a slim chance that the affected apps have some very specific Jpeg resources that trigger the crash, but more likely these libs were built with some wrong combination of flags and tools. For example, it could be that OPPO chose to use libjpeg while AOSP has switched to libjpeg-turbo some while ago.

I believe that for an individual developer who is haunted by such crash there is no real harm in setting minSdkVersion to 30, effectively opting out from the forced transition to 64-bit for the meanwhile. Methinks, such workaround should not be considered illegitimate on Play Store.

On the other hand, even before fixing the bug in Jpeg decoder, it is in power of Play Store to

track down such crashes across all apps
blacklist the devices that demonstrate such behavior so that they get a 32-bit bundle if available

DanAlbert · 2019-08-12T22:56:24Z

Then came @srikanthsunkari with Realme 3 Pro from the same manufacturer, but with API 28, so my suggestion happens to have been too optimistic.

It definitely looks to me like there's more than one bug being discussed here. Could be that your suggestion is correct for working around a subset of them.

DanAlbert · 2019-08-12T23:36:16Z

Found https://stackoverflow.com/q/57412918/632035, which is probably the other crash, and yeah, that's yet another different stack trace.

It is always possible that this is actually just a case of memory corruption in the app that is causing a segfault later on. ASan might help if that is the case. Don't have enough information to know if that's the problem or if this actually is a device bug.

srikanthsunkari · 2019-08-13T09:57:18Z

Hi, @DanAlbert @alexcohn I managed to pull crash report here from my device which happens randomly, can you conclude something from this report?
Thanks.

alexcohn · 2019-08-18T11:10:21Z

@srikanthsunkari It seems that the crash happens while the surface is not ready:

18:35:34.323 18685-18685/com.test.surfaceview E/Target BindViewHolder: called
18:35:34.800 18685-18898/com.test.surfaceview A/libc: Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x40 in tid 18898 (hwuiTask1), pid 18685 (teststaging)

compare this to

18:35:26.732 18685-18685/com.test.surfaceview E/Target BindViewHolder: called
18:35:27.107 18685-18685/com.test.surfaceview I/ExoPlayerImpl: Init 2.2.0 [tissot_sprout, Mi A1, Xiaomi, 28]
18:35:27.122 795-795/? D/SurfaceFlinger: duplicate layer name: changing SurfaceView - com.test.surfaceview/com.demo.test.ui.activities.MainActivity to SurfaceView - com.test.surfaceview/com.demo.test.ui.activities.MainActivity#2
18:35:27.124 795-795/? D/SurfaceFlinger: duplicate layer name: changing Background for -SurfaceView - com.test.surfaceview/com.demo.test.ui.activities.MainActivity to Background for -SurfaceView - com.test.surfaceview/com.demo.test.ui.activities.MainActivity#1
18:35:27.139 18685-20296/com.test.surfaceview I/OMXClient: IOmx service obtained

Also the name _ZZN7androidL46android_view_RenderNode_requestPositionUpdatesEP7_JNIEnvP8_jobjectlS3_EN26SurfaceViewPositionUpdater21doUpdatePositionAsyncEliiii suggests that a surfaceview position update failed here, and may not be directly caused by switch to 64-bit architecture.

srikanthsunkari · 2019-08-19T09:16:45Z

Thank you @alexcohn means issue may be due to my code changes? If so, since it is not reproducible any suggestion for this?

alexcohn · 2019-08-19T16:03:36Z

This could be some delicate time race, and it may be beyond your control. How often is this crash reproduced? How is it distributed across devices, OS versions, whatever? Crashlytics has a lot of helpful hints for you.

Actually, when you look at the full statistics, you can usually see that there are some crashes that have no rational explanation. If they affect less than 1 in 10000 active users, you can probably just ignore them (unless this 1 user happens to be your boss).

DanAlbert · 2019-08-20T20:23:08Z

Just to be sure: the switch to 64-bit is the only variable that changed when this crash started appearing? It wasn't the same release that also did something like switch to app bundles/APK splits, updated your NDK, switched compilers, or some other change? I'm guessing not, but someone asked so I wanted to clarify.

srikanthsunkari · 2019-08-21T07:14:26Z

This could be some delicate time race, and it may be beyond your control. How often is this crash reproduced? How is it distributed across devices, OS versions, whatever? Crashlytics has a lot of helpful hints for you.

After so many test cases, crash is always reproducible after we increased the SurfaceView instances > 6 in the layout, and leaving the SurfaceView tag without visibility tag.

srikanthsunkari · 2019-08-21T07:23:22Z

Just to be sure: the switch to 64-bit is the only variable that changed when this crash started appearing? It wasn't the same release that also did something like switch to app bundles/APK splits, updated your NDK, switched compilers, or some other change? I'm guessing not, but someone asked so I wanted to clarify.

Yes, we have had the change for the 64-bit flavor support, but no splits. And it is observed that only after this apk we have had this crash. And it happens only in OS 9.0 and above.
We have tested this in Samsung M20 (OS 9.0) manufacturer we have observed a reasonable error message saying below.

2019-08-13 17:26:32.364 4136-4136/com.test.surfaceview E/SurfaceView: A process tried to create too many surfaces
    android.view.WindowManager$TooManySurfacesException
        at android.view.SurfaceControl.nativeCreate(Native Method)
        at android.view.SurfaceControl.<init>(SurfaceControl.java:704)
        at android.view.SurfaceControl.<init>(SurfaceControl.java:74)
        at android.view.SurfaceControl$Builder.build(SurfaceControl.java:456)
        at android.view.SurfaceView$SurfaceControlWithBackground.<init>(SurfaceView.java:1474)
        at android.view.SurfaceView.updateSurface(SurfaceView.java:651)
        at android.view.SurfaceView$2.onPreDraw(SurfaceView.java:188)
        at android.view.ViewTreeObserver.dispatchOnPreDraw(ViewTreeObserver.java:991)
        at android.view.ViewRootImpl.performTraversals(ViewRootImpl.java:3069)
        at android.view.ViewRootImpl.doTraversal(ViewRootImpl.java:1942)
        at android.view.ViewRootImpl$TraversalRunnable.run(ViewRootImpl.java:8595)
        at android.view.Choreographer$CallbackRecord.run(Choreographer.java:988)
        at android.view.Choreographer.doCallbacks(Choreographer.java:765)
        at android.view.Choreographer.doFrame(Choreographer.java:700)
        at android.view.Choreographer$FrameDisplayEventReceiver.run(Choreographer.java:967)
        at android.os.Handler.handleCallback(Handler.java:873)
        at android.os.Handler.dispatchMessage(Handler.java:99)
        at android.os.Looper.loop(Looper.java:214)
        at android.app.ActivityThread.main(ActivityThread.java:7156)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:494)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:975)

alexcohn · 2019-08-21T07:38:12Z

@DanAlbert note that there are at least 3 distinct cases discussed in this thread

alexcohn · 2019-08-21T07:43:32Z

@srikanthsunkari it's not clear: on devices with OS version <9, you can open more surfaces than on Android 9, can't you? Also, on Android 9, you can open more surfaces when you force 32-bit bundle?

srikanthsunkari · 2019-08-21T10:21:00Z

on devices with OS version <9, you can open more surfaces than on Android 9, can't you?

This is not happening on below Android 9 (32-bit) devices.

Also, on Android 9, you can open more surfaces when you force 32-bit bundle?

Android 9 with 32-bit bundle APK also this crash produces.

DanAlbert · 2019-08-21T16:24:41Z

Given the error message it doesn't sound like the crash you're seeing is a bug, but just a limit you've exceeded.

srikanthsunkari · 2019-08-22T09:44:04Z

@DanAlbert would you confirm that this limit is from vendors implementation and not from the ndk? If it is from ndk it has to be handled and thrown a readable message from the android framework, not from ndk.

DanAlbert · 2019-08-27T17:18:31Z

I can guarantee it's not from the NDK. Your stack trace is Java.

DanAlbert · 2020-04-11T00:31:07Z

It doesn't seem like there's an NDK bug here, so closing. If that's wrong, lmk and we can reopen, but it sounds more like device bugs.

DanAlbert closed this as completed Apr 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crash in libjpeg.so when BitmapFactory.decodeStream called #1058

Crash in libjpeg.so when BitmapFactory.decodeStream called #1058

rauk123 commented Aug 6, 2019 •

edited by DanAlbert

Loading

alexcohn commented Aug 7, 2019

rauk123 commented Aug 8, 2019 •

edited by DanAlbert

Loading

alexcohn commented Aug 8, 2019

rauk123 commented Aug 8, 2019

alexcohn commented Aug 8, 2019 •

edited

Loading

rauk123 commented Aug 8, 2019

srikanthsunkari commented Aug 9, 2019

alexcohn commented Aug 11, 2019 •

edited

Loading

srikanthsunkari commented Aug 12, 2019

alexcohn commented Aug 12, 2019 via email

srikanthsunkari commented Aug 12, 2019

alexcohn commented Aug 12, 2019

DanAlbert commented Aug 12, 2019

DanAlbert commented Aug 12, 2019

alexcohn commented Aug 12, 2019

DanAlbert commented Aug 12, 2019

DanAlbert commented Aug 12, 2019

srikanthsunkari commented Aug 13, 2019 •

edited

Loading

alexcohn commented Aug 18, 2019

srikanthsunkari commented Aug 19, 2019

alexcohn commented Aug 19, 2019 •

edited

Loading

DanAlbert commented Aug 20, 2019 •

edited

Loading

srikanthsunkari commented Aug 21, 2019

srikanthsunkari commented Aug 21, 2019

alexcohn commented Aug 21, 2019

alexcohn commented Aug 21, 2019

srikanthsunkari commented Aug 21, 2019

DanAlbert commented Aug 21, 2019

srikanthsunkari commented Aug 22, 2019

DanAlbert commented Aug 27, 2019

DanAlbert commented Apr 11, 2020

Crash in libjpeg.so when BitmapFactory.decodeStream called #1058

Crash in libjpeg.so when BitmapFactory.decodeStream called #1058

Comments

rauk123 commented Aug 6, 2019 • edited by DanAlbert Loading

alexcohn commented Aug 7, 2019

rauk123 commented Aug 8, 2019 • edited by DanAlbert Loading

alexcohn commented Aug 8, 2019

rauk123 commented Aug 8, 2019

alexcohn commented Aug 8, 2019 • edited Loading

rauk123 commented Aug 8, 2019

srikanthsunkari commented Aug 9, 2019

alexcohn commented Aug 11, 2019 • edited Loading

srikanthsunkari commented Aug 12, 2019

alexcohn commented Aug 12, 2019 via email

srikanthsunkari commented Aug 12, 2019

alexcohn commented Aug 12, 2019

DanAlbert commented Aug 12, 2019

DanAlbert commented Aug 12, 2019

alexcohn commented Aug 12, 2019

DanAlbert commented Aug 12, 2019

DanAlbert commented Aug 12, 2019

srikanthsunkari commented Aug 13, 2019 • edited Loading

alexcohn commented Aug 18, 2019

srikanthsunkari commented Aug 19, 2019

alexcohn commented Aug 19, 2019 • edited Loading

DanAlbert commented Aug 20, 2019 • edited Loading

srikanthsunkari commented Aug 21, 2019

srikanthsunkari commented Aug 21, 2019

alexcohn commented Aug 21, 2019

alexcohn commented Aug 21, 2019

srikanthsunkari commented Aug 21, 2019

DanAlbert commented Aug 21, 2019

srikanthsunkari commented Aug 22, 2019

DanAlbert commented Aug 27, 2019

DanAlbert commented Apr 11, 2020

rauk123 commented Aug 6, 2019 •

edited by DanAlbert

Loading

rauk123 commented Aug 8, 2019 •

edited by DanAlbert

Loading

alexcohn commented Aug 8, 2019 •

edited

Loading

alexcohn commented Aug 11, 2019 •

edited

Loading

srikanthsunkari commented Aug 13, 2019 •

edited

Loading

alexcohn commented Aug 19, 2019 •

edited

Loading

DanAlbert commented Aug 20, 2019 •

edited

Loading