SIGTRAP Android freeze #1982
Comments
|
We would definitely need a reproducible example and also a sense of the APIs which are used (Reanimated version, Skia animations, etc) |
|
Hi @wcandillon, @LeviWilliams, We also experience this crash in production. So much - unfortunately - that we received a warning from Google regarding 'Android vitals bad behavior'. Our app will become less discoverable and receive a warning at the store page if this crash is not resolved soon. We currently have no clue when/where specifically this crash occurs. Therefore we cannot provide a reproducible example at the moment. For our app, in the last 7 days 1.5k users experienced 1.7k crashes of the SIGTRAP type, originated from Two crashes occur, both indicated by SIGTRAP, both about 50% of the total occurrences; and Some more details100% foreground crashes, spreaded to usage over Android versions and devices. Versions used in the context of the above; With Skia 0.1.210: No significant crashes Hope this helps tracking down the issue. |
|
@laurens-lamberts Thank you for this precious data.And I hope that we can get this sorted out as soon as possible. I need to review things more carefully on my side but from The error seems to be Skia specific, I will investigate this a bit deeper and let you know if I find anything. |
|
Strangely enough, I cannot find any relevant change from Going forward, we will setup some of release program to check if we introduce such regressions to releases. We have a RN Skia client of approximately the same scale as you running @laurens-lamberts I suggest we do the following:
|
|
Thanks a lot @wcandillon, we're really happy with your support proposal on this issue. Due to the high impact when issues arise during deployment in the christmas / newyear period, we will postpone next releases to January. For the upcoming release of our project we upgraded to the following library versions (all latest); If any new versions of the above packages appear before our release, we will update to ensure having the latest of all. We always perform our releases phased, so as soon as we got insight in crash rates we will share them with you. This will likely be the end of January / beginning of Februari. For my information, where in the react-native-skia library do I find the reference to the internal Skia version number (like |
|
This is were you can find the Skia version used : https://github.com/Shopify/react-native-skia/blob/main/.gitmodules I will continue to investigate this a bit and also do the upgrade to m121 and we can tackle this more aggressively after the holidays. I think that we are lucky to have a Skia client that uses 113 at scale but with only non-deprecated API, that will give us a lot of information once/when they deploy 114 and above. |
|
Yes, that's great. Looking forward to hearing their experiences with later versions. |
|
Any updates on this/ways to resolve it @laurens-lamberts @wcandillon @LeviWilliams ? Anything we can do to help? We just had to downgrade Skia to 196 because of thousands of crashes in production due to this error. Would be awesome to be able to upgrade since we want to move on to RN 0.73 (which according to release notes are not fully supported until 213) 🙏 |
|
@espenjanson Yes anything that would help to reproduce the issue or more details on the conditions of the crash would be extremely useful. I'm surprised you are on We have been coordinating with @laurens-lamberts to find the root cause of the issue but without success yet. We have a large client who's running the latest version of Skia without any crashes (this same client had a large amount of crashes in production with |
|
@espenjanson could you send me a list of Skia APIs and components you are using? You can do it privately as well by email. |
|
@wcandillon thanks for quick response. We'd love to help in any way we can. Any chance you could provide the package.json (or at least parts of it, such as react-native and reanimated version and perhaps other libraries that could affect skia)? If you want to, we can send you a minimal functioning app project with our crashing package.json and all the components we have that use Skia. Can put together a zip or a repo, whatever works better for you. If needed, we can also provide more detailed stack traces from Google Play and Sentry. Will put the team on this immediately. Thanks a million for paying attention to this! |
|
@wcandillon We have exactly same issue with quite big number of crashes with exact same error messages that started to My guess it's not possible to create 100% reproducible example because crash is quite random, but it's probably related to Skia + Animations. I will try to some app that uses same features as our production app and that will run that animations in some forever loop and hope it will crash after some time. Also I will try to just mount and unmount our components very quickly in some forever loop. I will let you know if I will find something. |
|
@Nodonisko is this on the latest version? It looks like this may have been fixed after the latest Skia version upgrade. |
|
Hi @wcandillon, We are live on 1.1.0 of react-native-skia and still experience the crash. Is this already using the latest Skia version? |
|
I'm slowly formulating a plan to tackle this issue. Is there a sense of which screen the crash is happening? This would allow us to funnel the API/code that might be faulty. |
|
@wcandillon We are not at latest version we have like two months old version. We will it but it will take us another month to test it in production. In mean time we were quite lucky and one of our testers managed to catch crash on video. It's not much helpful but at least we know on which screen it's happening. Sadly it's screen full of Skia components and animations :D I will try to prepare some standalone app from that screen so we can try to reproduce it in more isolated env. Hope I will have this done today or tmrw. screen-20240527-115456.mp4 |
|
nice, is this happening in debug mode? is there more details maybe
when the crash happens?
…On Tue, May 28, 2024 at 1:57 PM Daniel Suchý ***@***.***> wrote:
@wcandillon We are not at latest version we have like two months old version. We will it but it will take us another month to test it in production.
In mean time we were quite lucky and one of our testers managed to catch crash on video. It's not much helpful but at least we know on which screen it's happening. Sadly it's screen full of Skia components and animations :D
I will try to prepare some standalone app from that screen so we can try to reproduce it in more isolated env. Hope I will have this done today or tmrw.
https://github.com/Shopify/react-native-skia/assets/5837757/6ff2b12f-9c2a-4af3-a1ff-75cd84957560
—
Reply to this email directly, view it on GitHub or unsubscribe.
You are receiving this email because you were mentioned.
Triage notifications on the go with GitHub Mobile for iOS or Android.
|
|
Sadly it's production version. |
|
So I spend most of yesterday trying to reproduce the issue. I created special version of our homescreen that runs animations in loop, mounting unmounting components etc. and let it run on two different Android devices for 30 minutes few times. I also tried both production and debug builds. So far I did not get single crash... I also noticed that there is one single crash in our GPlay console that is happening with nearly same signature I am not sure if it's related or it's some completely unrelated random crash with similar signature, but it's suspicious. Any ideas what we can try next? We can try to deploy some special version of Skia from @wcandillon if that could help in our next release, but I will take another month to roll this out to our users. I will also try to narrow in which version crash occurred first. |
|
So it seems that crash occurred after update from Also found this issue in Skia issue trackers https://issues.skia.org/issues/333423686 |
|
@Nodonisko Does https://issues.skia.org/issues/333423686 look like it could be related? It wasn't clear to me looking the bug report @Nodonisko @laurens-lamberts @LeviWilliams I would like to find some way to reproduce the issue (even in release mode) and/or pin down the scenario in which the error happen. I wrote this example that stresses the Skia APIs and I tested also in release mode: https://github.com/user-attachments/files/15758252/StressTest.zip Please let me know how I could update the test scenario to better match your circumstances. In #2396, we are experiencing a clear race condition which we are currently investigating and that might shed some light on what is happening. |
|
@wcandillon Error in https://issues.skia.org/issues/333423686 looks very similar to our error but not sure it's related. About stress test, I run it on my two devices in debug mode and no crash so far. I also went through Sentry data and it seems it crashes very often (not exclusively) when user goes to new screen, both when previous screen is unmounted (like tab change) or also if new screen is pushed into stack, which leads me to idea that this probably happens when some Skia component is mounted. |
|
We just released new version of app with 1.3.7 Skia version and this issue still persist. My colleague just got this crash when he did some hover gesture over our graph (Revolut style graph animation). |
|
Hello @wcandillon @Nodonisko ! Any updates here? We're still experiencing both issues on react-native-skia version 1.3.9. Any workaround or solution to help us mitigate these would be greatly appreciated, as we’re currently seeing hundreds of crashes related to both problems Could we consider downgrading to version 196/210? @espenjanson /@laurens-lamberts mentioned in this thread that it helped them, though I don’t see this as a sustainable long-term fix, even if it does work. (Also, @espenjanson / @laurens-lamberts , did it actually resolve the issues? Have you encountered any more problems since?) other relevant information: Skia API used:
These APIs were used on the latest release, the one where we started seeing the crashes. We created a new component that is used on several lists (FlashList, FlatList, SectionList). Each list has somewhere around 10-50 items
|
|
We are observing the same pair of crashes reported above. Like other commenters above, the crash is rare enough that we have not been able to reproduce it consistently (or at all?) during development. react-native-skia has been a joy to program with, but this crash is a bit of a show stopper as it is happening frequently enough to raise our crash rate above the "bad behaviour" threshold in Google Play. We have been running react-native-skia version 1.3.13 and the issue persists after rolling out an optimistic upgrade to 1.5.2. A downgrade to 0.1.210 (a crash-less version reported above) is not possible without considerable effort to replace our use of newer react-native-skia features like ParagraphBuilder. |
|
@tmgrask We are actually working on a fix for this at the moment. We found a serious threading issue on Android which we are working on. These crashes are hard to reproduce so I cannot guarantee 100%. However the indication that |
|
The |
|
I am working on subsequent improvements that I expect will improve the
situation. However, is there any chance you can show me the kind of
Skia usage is causing those crashes? As well as API version numbers.
It would be very useful if I could find ways to reproduce these
errors.
|
|
Our app uses many react-native-skia components and APIs, but there is really just 1 main screen where they are all combined, so I cannot pinpoint a specific one. Many of the react-native-skia components are animated via react-native-reanimated. From the videos of the fuzz testing, the crashes are not happening at the same time/with the same elements in view. I have also been trying to reproduce the issue in a controlled environment with no success. I will let you know if I stumble on any more relevant info. Through google play console I can see the SIGTRAP crashes are happening in producton on Android versions 8 through 14, so android API versions 28->34. I don't think we have enough data to say 35 is not also effected. |
|
v1.5.7 is published, I would love to get a read on whether it is improving Android stability? |
|
From early testing it looks like you have solved the new (#2754 ) crash in 1.5.7, nice work! Unfortunately, I still see the SIGTRAP crash in fuzz testing. I have been unable to find a way to reproduce locally but I'm still looking. |
|
Continuing to investigate the issue, #2761 will help a lot as well. I'm curious to see if it reduces the crashes. However if your app is crossing thread boundaries, you will now get null for some images in places where you might have gotten a result (but setting the direct context to a bad state). |
|
@wcandillon I am able to very inconsistently reproduce the crash in our app by running the Android exerciser monkey against a release build on an emulator. I almost hesitate to share this with you because I worry it will waste your time to chase this very rare repro. Our app source code is now public, I made a branch with the monkey script that (rarely) causes the crash: https://github.com/Psiphon-Inc/conduit/tree/skia-crash-repro The app uses react-native-skia in many places, but steps for very rare repro:
I just ran it 10 times and got it once, on the 7th time: I have yet to see it work from an Not exactly the most reproducible reproduction! |
|
Thank you @tmgrask this is huge. I'm am trying it now. On you side, if you use |
|
@tmgrask I was able to reproduce the crash but now trying to reproduce it with debug symbols and maybe adding some try/catches at the jni layer to see if some unhandled exception happens. I will publish 1.5.9 also which improve stability on Android. Any chance you could contribute the orb as a standalone example in https://github.com/Shopify/react-native-skia/tree/main/apps/paper/src? I didn't know about this monkey command but it's very useful :) I want to add it to the CI on both Skia and also RN WebGPU. Hope we can get to the bottom of this quickly. |
|
I tried Upgrading to 1.5.9 things look a little different: Here is a new crash I just saw locally (
When I put an APK with 1.5.9 through device farm fuzz test, there is another new crash that happens quite a bit. The backtrace doesn't explicitly mention rnskia, so perhaps a red herring, but it is noteworthy that I do not observe this crash in a build with 1.5.8. The number of SIGTRAP crashes is down (but still happening), however I have a feeling the SIGABRT crash above may be masking it, since the SIGABRT crash is happening relatively frequently. |
|
@tmgrask I ran the monkey script 10 times no crash, that's a good sign no? On previous version I feel like could reproduce it pretty quickly? |
|
@wcandillon 1.5.10 looks way better! Overall, device farm fuzz SIGTRAP rate is way down:
I am unsure if the SIGABRT EGL_BAD_ALLOC error is relevant, but it does reference I did repro SIGTRAP locally once, so it is still possible, but it does indeed seem much less common. This improvement is enough for me to ship an update to users, to hopefully alleviate our Google Play crash rate issue. SIGABRT (EGL_BAD_ALLOC):
Thanks! Your videos from beautiful Zurich Switzerland were very helpful! |
|
This is very exciting. Is it possible that the I love the monkey script, I really want to add it to the CI. if you want to contribute an example to the example app that would get tested there that would be fun. |
|
@wcandillon Thanks for fixing it, we are going to try it ASAP. Can you please also share that |
|
Unfortunately, we can still replicate this issue in version 1.5.10. We haven’t identified a clear, consistent pattern for reproducing it, other than the fact that it occurs when rendering canvases within a SectionList. Here is the stack trace we’ve captured: We also got this stacktrace which I think is related to something different, but still reproduced when scrolling throughout the SectionList with canvases |
|
@alexnaiman any chance you could upgrade to v1.7.4 or higher? |
|
@wcandillon We could try updating to a newer version. I will do that and come back later with the results. It's really hard to us to reproduce :( Regarding the devices, we reproduced the issues on:
|
|
This is useful info, in v1.7.2 we fixed an issue that was happening on samsung devices only (as far as we know). The bug was related to devices that use Angle on top of Vulkan for OpenGL support. If the issue still happens, there are way we could try to track it down. When you are trying to reproduce the issue, are you using one of these devices? (using browserstack for instance). |
|
I've spotted a couple of SIGTRAP errors in our logs. This one has specifically affected Samsung devices (but I don't have enough data to be certain it's not a coincidence): https://gotterdammerung.sentry.io/share/issue/4484190c4ddf45cba115cc477bfeb76c/ Here's another one, potentially related, (but I honestly have no idea if it is): https://gotterdammerung.sentry.io/share/issue/ac18d876545648378c1696a67374660f/ And a third one that was caught by Google Play Console's testing. It happens right at the moment of rendering a button (which is rendered through Skia) in a scrollview:
Full logs: logcat (1).txt This has occurred on either 1.5.0, 1.5.10 or 1.7.1. Will ofc create separate issues when/if I manage to reproduce these. I can't say for certain if they're all even related to Skia because there's so little information in the logs for some of them. |
|
Yes Adam, that would be great, especially since you seem to have hit
the bug we fixed for Samsung devices, here is the patch:
https://github.com/Shopify/react-native-skia/compare/main...wcandillon:react-native-skia:android-resize?expand=1
…On Mon, Dec 23, 2024 at 3:56 PM Adam Gerthel ***@***.***> wrote:
I've spotted a couple of SIGTRAP errors in our logs. This one has specifically affected Samsung devices (but I don't have enough data to be certain it's not a coincidence): https://gotterdammerung.sentry.io/share/issue/4484190c4ddf45cba115cc477bfeb76c/
Here's another one, potentially related, (but I honestly have no idea if it is): https://gotterdammerung.sentry.io/share/issue/ac18d876545648378c1696a67374660f/
Will ofc create separate issues when/if I manage to reproduce these.
—
Reply to this email directly, view it on GitHub or unsubscribe.
You are receiving this email because you commented on the thread.
Triage notifications on the go with GitHub Mobile for iOS or Android.
|
|
the second crash you are showing, we are working on a fix for it as
well, it's a bit more work on our side before publishing it.
William
…On Mon, Dec 23, 2024 at 4:44 PM William Candillon ***@***.***> wrote:
Yes Adam, that would be great, especially since you seem to have hit
the bug we fixed for Samsung devices, here is the patch:
https://github.com/Shopify/react-native-skia/compare/main...wcandillon:react-native-skia:android-resize?expand=1
On Mon, Dec 23, 2024 at 3:56 PM Adam Gerthel ***@***.***> wrote:
>
> I've spotted a couple of SIGTRAP errors in our logs. This one has specifically affected Samsung devices (but I don't have enough data to be certain it's not a coincidence): https://gotterdammerung.sentry.io/share/issue/4484190c4ddf45cba115cc477bfeb76c/
>
> Here's another one, potentially related, (but I honestly have no idea if it is): https://gotterdammerung.sentry.io/share/issue/ac18d876545648378c1696a67374660f/
>
> Will ofc create separate issues when/if I manage to reproduce these.
>
> —
> Reply to this email directly, view it on GitHub or unsubscribe.
> You are receiving this email because you commented on the thread.
>
> Triage notifications on the go with GitHub Mobile for iOS or Android.
>
>
|
We managed to reproduce the issues only on those devices (physical devices), and we couldn't figure out a pattern. I will do the update and see if it changes anything I will come back with the results |
Is patch mentioned here already merged? We are still getting crashes in |
AFAIK, that patch was merged and released in 1.7.7, see #2840 in https://github.com/Shopify/react-native-skia/releases/tag/v1.7.7. But I don't think that's related to your issues since that bug started in 1.7.2 and you're seeing issues on 1.5.10. |
|
@AdamGerthel Thanks for explanation, I was little bit lost since there is already quite a lot of posts and we are hunting this bug for more than year now :D |
|
Hello Daniel 👋, to give a bit of insight on what the bug was, on
devices that use Vulkan (using Angle for OpenGL support) which is a
set of Samsung devices, the following flushing needed to be executed
to disconnect the window api properly:
https://github.com/wcandillon/react-native-skia/blob/main/packages/skia/android/cpp/rnskia-android/OpenGLWindowContext.h#L60
I found the answer on stackoverflow, I would have never found this by
myself 🙈 Very niche issue.
…On Sun, Jan 5, 2025 at 8:05 PM Daniel Suchý ***@***.***> wrote:
@AdamGerthel Thanks for explanation, I was little bit lost since there is already quite a lot of posts and we are hunting this bug for more than year now :D
—
Reply to this email directly, view it on GitHub or unsubscribe.
You are receiving this email because you commented on the thread.
Triage notifications on the go with GitHub Mobile for iOS or Android.
|
Description
Recently we upgraded our Skia version from 0.1.197 to 0.1.214 and now we are seeing a bunch of "SIGTRAP: Trace/breakpoint trap" in production on Android devices with a variety of versions. Currently on react-native 0.72.3 if helpful.
.
We reproduced a couple times on a Pixel 6 and the app just freezes. I understand this error is vague, we currently have no leads as to why this happens after updating the lib though if we are able to provide a repro we will.
Let us know if there are any ideas on what we can try, thanks for the help as always.
Version
0.1.214
Steps to reproduce
Snack, code example, screenshot, or link to a repository
The text was updated successfully, but these errors were encountered: