Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Android app crashing #5757

Closed
SteveScott opened this issue Sep 5, 2023 · 15 comments
Closed

Android app crashing #5757

SteveScott opened this issue Sep 5, 2023 · 15 comments
Labels
a-Android P0 critical Highest priority severe: crash The app quits, or stops responding. upstream: other Issues related to an issue in another dependency

Comments

@SteveScott
Copy link

For the past month, as I am typing on my Android Pixel 7 Pro, Zulip crashes. No message or notification. I re-load the app and my text is still there. What can I do to help troubleshoot or report this bug? Version 27.211.

@chrisbobbe
Copy link
Contributor

Hi @SteveScott, thanks for the report! What version of Android are you on?

@SteveScott
Copy link
Author

SteveScott commented Sep 5, 2023 via email

@chrisbobbe
Copy link
Contributor

Hmm, I wonder if this is #5741.

@kylehousley
Copy link

I have also been experiencing the same issue for the past couple weeks, on Android 13 as well.

@chrisbobbe
Copy link
Contributor

We're aware that Android crashes have been becoming much more frequent recently, especially on Google Pixel devices, and we're investigating the cause.

@maltokyo
Copy link

I also have this, started recently (in last two or three days) on Xiaomi phone running latest version of Android 13. Crashes now every time I open the app. How can I get logs to share, I can't work it out.

Tried deleting cache and all app data and logging in again, same issue.

@lakinwecker
Copy link

I've been experiencing crashes for the past 2ish weeks as well, Android 13, almost always when typing, sometimes when not. Other lichess team members are also all starting to experience the same crashes as well.

@L-Marcel
Copy link

I also have this, started recently (in last two or three days) on Xiaomi phone running latest version of Android 13. Crashes now every time I open the app. How can I get logs to share, I can't work it out.

Tried deleting cache and all app data and logging in again, same issue.

Me too.

@chrisbobbe chrisbobbe added P0 critical Highest priority severe: crash The app quits, or stops responding. labels Sep 19, 2023
@chrisbobbe
Copy link
Contributor

chrisbobbe commented Sep 19, 2023

Thank you all for your patience. Debugging continues in the Zulip development community, here.

Quoting @gnprice from there (link):

For users who are technically sophisticated and willing to help debug, I think probably the most useful thing is to use adb logcat to get a crash dump.

Note that the Android logs have a "crash" section which is separate from the main section. The stack trace is likely to be in the "crash" section, and is the most interesting piece to get.

Those would help us see if we're seeing the same stack trace all the time, or several of them. And the stack traces might have something in them that helps identify a piece of the system that's likely to be involved.

The other most likely fruitful path to pursue is probably to study the [Google Play system update changes] to see what happened between the July and August updates that might plausibly trigger something like this.

Then that information can feed into that [thread] in the Android tracker, and may help get it more quickly routed to the right team, or more quickly diagnosed once it is, and then fixed.

Knowing the system involved, or a diagnosis, might also give us an idea of something we could do to avoid triggering the issue.

Or it might give an idea of something [React Native] could do to avoid triggering it.

We suspect that a Google Play system update from August 1 is a factor in this crash. It seems those updates are rolled out gradually. If you tap Settings > About phone > Android version, you should see two dates, like this:

Android security update: August 5, 2023
Google Play system update: August 1, 2023

For anyone who's been seeing this issue's symptoms—crashes that started or became frequent in the last week or two—you may want to check that "Google Play system update" date. We expect you'll see August 1 there, and we'd like to know if that's wrong. Also, we don't yet have reports from anyone on a September 1 update, and we'd like to know if these new crashes persist with that update.

@russp41
Copy link

russp41 commented Sep 20, 2023

I've been experiencing this issue since Sept 14. Zulip crashes frequently on Samsung Galaxy S23 Ultra, Android 13, "Google Play system update" date is August 1.

@hendersonreed
Copy link

chiming in: I've been seeing crashes for the past week at least, primarily after tapping on push notifications or typing out messages. Google Play system update is from August 1.

@RodEsp
Copy link

RodEsp commented Sep 20, 2023

Same as above, been having random crashes for a few weeks on a Pixel 7 Pro.
Google Play system update is August 1 too .

Here's a crash log I got, from two crashes that happened last night.

adb logcat --buffer=crash
09-19 21:27:40.322 25796 25938 F libc    :
09-19 21:27:40.322 25796 25938 F libc    : stack corruption detected (-fstack-protector)
09-19 21:27:40.334 25796 25796 F libc    : stack corruption detected (-fstack-protector)
09-19 21:27:40.803 25796 25796 F libc    : Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 25796 (com.zulipmobile), pid 25796 (com.zulipmobile)
09-19 21:27:41.883 26127 26127 F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
09-19 21:27:41.883 26127 26127 F DEBUG   : Build fingerprint: 'google/cheetah/cheetah:13/TQ3A.230805.001/10316531:user/release-keys'
09-19 21:27:41.883 26127 26127 F DEBUG   : Revision: 'MP1.0'
09-19 21:27:41.883 26127 26127 F DEBUG   : ABI: 'arm64'
09-19 21:27:41.883 26127 26127 F DEBUG   : Timestamp: 2023-09-19 21:27:41.015024983-0400
09-19 21:27:41.883 26127 26127 F DEBUG   : Process uptime: 16s
09-19 21:27:41.883 26127 26127 F DEBUG   : Cmdline: com.zulipmobile
09-19 21:27:41.883 26127 26127 F DEBUG   : pid: 25796, tid: 25796, name: com.zulipmobile  >>> com.zulipmobile <<<
09-19 21:27:41.883 26127 26127 F DEBUG   : uid: 10440
09-19 21:27:41.883 26127 26127 F DEBUG   : tagged_addr_ctrl: 0000000000000001 (PR_TAGGED_ADDR_ENABLE)
09-19 21:27:41.883 26127 26127 F DEBUG   : signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
09-19 21:27:41.883 26127 26127 F DEBUG   :     x0  0000000000000000  x1  00000000000064c4  x2  0000000000000006  x3  b400007d500da680
09-19 21:27:41.883 26127 26127 F DEBUG   :     x4  607372652c271f63  x5  607372652c271f63  x6  607372652c271f63  x7  7f7f7f7f7f7f7f7f
09-19 21:27:41.883 26127 26127 F DEBUG   :     x8  00000000000000f0  x9  0000007deb9179e0  x10 0000000000000001  x11 0000007deb959370
09-19 21:27:41.883 26127 26127 F DEBUG   :     x12 0000007e0626e020  x13 000000007fffffff  x14 0000000008c22b82  x15 000007e8d97b033c
09-19 21:27:41.883 26127 26127 F DEBUG   :     x16 0000007deb9c6d50  x17 0000007deb9a1eb0  x18 0000007e05afc000  x19 00000000000064c4
09-19 21:27:41.883 26127 26127 F DEBUG   :     x20 00000000000064c4  x21 00000000ffffffff  x22 0000007e04f9e000  x23 b400007d400f3090
09-19 21:27:41.883 26127 26127 F DEBUG   :     x24 b400007d400f3090  x25 0000007e04f9e000  x26 0000007ad25d0f20  x27 000000000000000b
09-19 21:27:41.883 26127 26127 F DEBUG   :     x28 0000000008000004  x29 b400007d500da700
09-19 21:27:41.883 26127 26127 F DEBUG   :     lr  0000007deb9491c8  sp  b400007d500da660  pc  0000007deb9491f4  pst 0000000000001000
09-19 21:27:41.883 26127 26127 F DEBUG   : backtrace:
09-19 21:27:41.883 26127 26127 F DEBUG   :       #00 pc 00000000000531f4  /apex/com.android.runtime/lib64/bionic/libc.so (abort+164) (BuildId: dc4001c2ef2dfc23467040797a96840c)
09-19 21:27:41.883 26127 26127 F DEBUG   :       getsentry/sentry-unity#1 pc 0002b1970002b192  <unknown>
09-20 00:59:51.237 26157 26206 F libc    : stack corruption detected (-fstack-protector)
09-20 00:59:51.238 26157 26206 F libc    : stack corruption detected (-fstack-protector)
09-20 00:59:51.425 26157 26206 F libc    : Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 26206 (mqt_native_modu), pid 26157 (com.zulipmobile)
09-20 00:59:52.014 22770 22770 F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
09-20 00:59:52.015 22770 22770 F DEBUG   : Build fingerprint: 'google/cheetah/cheetah:13/TQ3A.230805.001/10316531:user/release-keys'
09-20 00:59:52.015 22770 22770 F DEBUG   : Revision: 'MP1.0'
09-20 00:59:52.015 22770 22770 F DEBUG   : ABI: 'arm64'
09-20 00:59:52.015 22770 22770 F DEBUG   : Timestamp: 2023-09-20 00:59:51.654086572-0400
09-20 00:59:52.015 22770 22770 F DEBUG   : Process uptime: 12728s
09-20 00:59:52.015 22770 22770 F DEBUG   : Cmdline: com.zulipmobile
09-20 00:59:52.015 22770 22770 F DEBUG   : pid: 26157, tid: 26206, name: mqt_native_modu  >>> com.zulipmobile <<<
09-20 00:59:52.015 22770 22770 F DEBUG   : uid: 10440
09-20 00:59:52.015 22770 22770 F DEBUG   : tagged_addr_ctrl: 0000000000000001 (PR_TAGGED_ADDR_ENABLE)
09-20 00:59:52.015 22770 22770 F DEBUG   : signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
09-20 00:59:52.015 22770 22770 F DEBUG   : Abort message: 'pdata'
09-20 00:59:52.015 22770 22770 F DEBUG   :     x0  0000000000000000  x1  000000000000665e  x2  0000000000000006  x3  b400007d500ca5e0
09-20 00:59:52.015 22770 22770 F DEBUG   :     x4  607372652c271f63  x5  607372652c271f63  x6  607372652c271f63  x7  7f7f7f7f7f7f7f7f
09-20 00:59:52.015 22770 22770 F DEBUG   :     x8  00000000000000f0  x9  0000007deb9179e0  x10 0000000000000001  x11 0000007deb959370
09-20 00:59:52.015 22770 22770 F DEBUG   :     x12 0000007e0626e020  x13 000000007fffffff  x14 00000000090147fe  x15 00000831b244021d
09-20 00:59:52.015 22770 22770 F DEBUG   :     x16 0000007deb9c6d50  x17 0000007deb9a1eb0  x18 0000007abd504000  x19 000000000000662d
09-20 00:59:52.015 22770 22770 F DEBUG   :     x20 000000000000665e  x21 00000000ffffffff  x22 0000000000000400  x23 0000007abd917000
09-20 00:59:52.015 22770 22770 F DEBUG   :     x24 b400007d500ca6f8  x25 0000007e04f9e000  x26 0000007dff28f358  x27 0000007dff2909b0
09-20 00:59:52.015 22770 22770 F DEBUG   :     x28 0000007dff28f748  x29 b400007d500ca660
09-20 00:59:52.015 22770 22770 F DEBUG   :     lr  0000007deb9491c8  sp  b400007d500ca5c0  pc  0000007deb9491f4  pst 0000000000001000
09-20 00:59:52.015 22770 22770 F DEBUG   : backtrace:
09-20 00:59:52.015 22770 22770 F DEBUG   :       #00 pc 00000000000531f4  /apex/com.android.runtime/lib64/bionic/libc.so (abort+164) (BuildId: dc4001c2ef2dfc23467040797a96840c)
09-20 00:59:52.015 22770 22770 F DEBUG   :       getsentry/sentry-unity#1 pc 000000000006a3e4  /apex/com.android.runtime/lib64/bionic/libc.so (__stack_chk_fail+20) (BuildId: dc4001c2ef2dfc23467040797a96840c)
09-20 00:59:52.015 22770 22770 F DEBUG   :       getsentry/sentry-unity#2 pc 000000000004e264  /apex/com.android.runtime/lib64/bionic/libc.so (async_safe_fatal_va_list+436) (BuildId: dc4001c2ef2dfc23467040797a96840c)
09-20 00:59:52.015 22770 22770 F DEBUG   :       getsentry/sentry-unity#3 pc b400007d500cae30  <unknown>

@quinn-dougherty
Copy link

Galaxy S23, new. Unusably bad crashing.

Android 13
Google Play System Update Aug 1

chrisbobbe added a commit to chrisbobbe/zulip-mobile that referenced this issue Sep 21, 2023
Our hope here is to fix the Android crashes in zulip#5757. If it works
(as it seems to in internal testing!), it would support the theory
that the Sentry Native SDK at 0.4.12 (and from others' reports, also
0.5.4) has a bad interaction with something in Google's Play
Services system update on 2023-08-01 -- and that the interaction is
not present with Sentry Native 0.6.5 (its current latest).

Those Sentry Native SDK versions are deduced from Sentry *React
Native* SDK versions (3.4.3, 5.1.1, and 5.9.2, respectively), using
a process I wrote down:
  https://chat.zulip.org/#narrow/stream/48-mobile/topic/Android.20crashes/near/1643817

See a React Native issue:
  facebook/react-native#39505

and an issue in a Google tracker:
  https://issuetracker.google.com/issues/300840851?pli=1

and an issue in the Sentry Unity tracker:
  https://github.com/getsentry/sentry-unity/issues/1444

Done by following the upgrade guide at
  https://docs.sentry.io/platforms/react-native/migration/
starting in the "From 3.x to 4.x" section and working up through
"From 4.x to 5.x".

Not many declared breaking changes relevant to our app:
- The `Severity` enum is removed in favor of string literals
- The Sentry CLI had a big upgrade with declared breaking changes
  to handle in our iOS build

We want to get this out to Android users quickly, so we postpone a
few things for now:
- Any testing on iOS
- Flow types generated from the TypeScript using TsFlower. (We have
  a hand-translated libdef, but it's likely to be misleading because
  it's now outdated by two major versions. I've ripped it out here.)

See our long debugging thread on CZO:
  https://chat.zulip.org/#narrow/stream/48-mobile/topic/.40sentry.2Freact-native.20to.20latest/near/1643986
chrisbobbe added a commit to chrisbobbe/zulip-mobile that referenced this issue Sep 21, 2023
Our hope here is to fix the Android crashes in zulip#5757. If it works
(as it seems to in internal testing!), it would support the theory
that the Sentry Native SDK at 0.4.12 (and from others' reports, also
0.5.4) has a bad interaction with something in Google's Play
Services system update on 2023-08-01 -- and that the interaction is
not present with Sentry Native 0.6.5 (its current latest).

Those Sentry Native SDK versions are deduced from Sentry *React
Native* SDK versions (3.4.3, 5.1.1, and 5.9.2, respectively), using
a process I wrote down:
  https://chat.zulip.org/#narrow/stream/48-mobile/topic/Android.20crashes/near/1643817

See a React Native issue:
  facebook/react-native#39505

and an issue in a Google tracker:
  https://issuetracker.google.com/issues/300840851?pli=1

and an issue in the Sentry Unity tracker:
  https://github.com/getsentry/sentry-unity/issues/1444

Done by following the upgrade guide at
  https://docs.sentry.io/platforms/react-native/migration/
starting in the "From 3.x to 4.x" section and working up through
"From 4.x to 5.x".

Not many declared breaking changes relevant to our app:
- The `Severity` enum is removed in favor of string literals
- The Sentry CLI had a big upgrade with declared breaking changes
  to handle in our iOS build

We want to get this out to Android users quickly, so we postpone a
few things for now:
- Any testing on iOS
- Flow types generated from the TypeScript using TsFlower. (We have
  a hand-translated libdef, but it's likely to be misleading because
  it's now outdated by two major versions. I've ripped it out here.)

See our long debugging thread on CZO:
  https://chat.zulip.org/#narrow/stream/48-mobile/topic/.40sentry.2Freact-native.20to.20latest/near/1643986

Fixes: zulip#5757
chrisbobbe added a commit to chrisbobbe/zulip-mobile that referenced this issue Sep 21, 2023
TODO:
- Testing on iOS
- Flow types generated from the TypeScript using TsFlower? (We have
  a hand-translated libdef, but it's likely to be misleading because
  it's now outdated by two major versions. I've ripped it out here.)

Our hope here was to fix the Android crashes in zulip#5757. It didn't
work; testing showed that the crash still happens on this later
version:
  https://chat.zulip.org/#narrow/stream/48-mobile/topic/Android.20crashes/near/1644202

So if there's a bad interaction between the Sentry Native SDK and
Google's Play Services system update on 2023-08-01, it's still
present with Sentry Native 0.6.5 (the current latest), as well as
other versions in prior reports: 0.4.12 (ours, just before this
commit) and 0.5.4.

For how to deduce a Sentry Native SDK version from a Sentry *React
Native* SDK version, see
  https://chat.zulip.org/#narrow/stream/48-mobile/topic/Android.20crashes/near/1643817

See a React Native issue:
  facebook/react-native#39505

and an issue in a Google tracker:
  https://issuetracker.google.com/issues/300840851?pli=1

and an issue in the Sentry Unity tracker:
  https://github.com/getsentry/sentry-unity/issues/1444

Still, good to do the upgrade anyway, and if Sentry comes out with a
fix, we'll be best placed to take it having already upgraded to the
current latest.

Done by following the upgrade guide at
  https://docs.sentry.io/platforms/react-native/migration/
starting in the "From 3.x to 4.x" section and working up through
"From 4.x to 5.x".

Not many declared breaking changes relevant to our app:
- The `Severity` enum is removed in favor of string literals
- The Sentry CLI had a big upgrade with declared breaking changes
  to handle in our iOS build

See our long debugging thread on CZO:
  https://chat.zulip.org/#narrow/stream/48-mobile/topic/.40sentry.2Freact-native.20to.20latest/near/1643986
@gnprice
Copy link
Member

gnprice commented Sep 21, 2023

Thanks everyone for the reports!

We've investigated this intensively over the past few days, and we've just released a version of Zulip that fixes these crashes: v27.213. If you're in the beta, the update is available now; if not, it'll be available once Play Store review is complete, which typically takes under a day.

For more details, read on.


This turns out to be an issue that's appearing in a number of different apps by different developers. It's being investigated in the issue trackers for Sentry, which is an error-reporting service we use, and for Android upstream:

The conditions that trigger the bug appear to be:

  • The device has taken the Google Play system update dated August 2023.
    • This is the part that's new — it seems like this update has only started rolling out widely in the past week.
    • There's also a September 2023 version, but it appears to trigger the bug the same as the August version. edit: I misread a report — it turns out I'm not aware of anyone running a Google Play system update later than the August one.
  • The device is running Android 12, 13, or 14; not Android 11 or older.
  • The app is using Sentry.

It's not yet clear whether the bug is in Sentry or in the Android system components that are updated in the Google Play system update. The issue persists even with the latest versions of the Sentry libraries, as we determined at #5765.

In order to make these crashes stop happening for everyone using Zulip, our v27.213 release simply disables Sentry on Android. This isn't a super comfortable state to be in — error reporting is important for us to learn of other issues and fix them — but it lets us resolve the crashes while the Sentry folks work on a diagnosis and fix. I'll file a follow-up issue (→ #5766) for re-enabling Sentry without the crashes.

@gnprice gnprice closed this as completed Sep 21, 2023
@gnprice gnprice added the upstream: other Issues related to an issue in another dependency label Sep 21, 2023
@chrisbobbe
Copy link
Contributor

If you see a crash after updating to v27.213, please open a new issue.

gnprice added a commit to gnprice/zulip-mobile that referenced this issue Sep 26, 2023
This is a workaround for zulip#5757, the crashes we've started to
regularly see as many users' devices have started taking the
Google Play system update dated 2023-08.

See the (long) chat thread here:
  https://chat.zulip.org/#narrow/stream/48-mobile/topic/Android.20crashes/near/1640893

and issue threads reported in Android's and Sentry's trackers:
  https://issuetracker.google.com/issues/300840851
  https://github.com/getsentry/sentry-unity/issues/1444

Our issue for re-enabling Sentry on Android is zulip#5766.

This commit was cherry-picked from 29cea89, part of v27.213,
with tweaks to mention zulip#5766.
chrisbobbe pushed a commit that referenced this issue Sep 26, 2023
This is a workaround for #5757, the crashes we've started to
regularly see as many users' devices have started taking the
Google Play system update dated 2023-08.

See the (long) chat thread here:
  https://chat.zulip.org/#narrow/stream/48-mobile/topic/Android.20crashes/near/1640893

and issue threads reported in Android's and Sentry's trackers:
  https://issuetracker.google.com/issues/300840851
  https://github.com/getsentry/sentry-unity/issues/1444

Our issue for re-enabling Sentry on Android is #5766.

This commit was cherry-picked from 29cea89, part of v27.213,
with tweaks to mention #5766.
@chrisbobbe chrisbobbe unpinned this issue Oct 10, 2023
gnprice added a commit that referenced this issue Oct 10, 2023
This reverts commit f792c39, and effectively reinstates
commit e8c57e7.  It's aimed as a workaround for #5778,
which appears to revive #5757.  See chat thread:
  https://chat.zulip.org/#narrow/stream/48-mobile/topic/Android.20crash.20.23M5778/near/1659732
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a-Android P0 critical Highest priority severe: crash The app quits, or stops responding. upstream: other Issues related to an issue in another dependency
Projects
None yet
Development

No branches or pull requests