[Bug] Firestore Android using too many global refs

[a-maurice]

[REQUIRED] Please fill in the following fields:

• Unity editor version: 2021.3.13f1
• Firebase Unity SDK version: 10.2.0
• Source you installed the SDK: unitypackage
• Problematic Firebase Component: Firestore
• Other Firebase Components in use: Crashlytics
• Additional SDKs you are using: N/A
• Platform you are using the Unity editor on: All
• Platform you are targeting: Android
• Scripting Runtime: N/A

[REQUIRED] Please describe the issue here:

(Please list the full steps to reproduce the issue. Include device logs, Unity logs, and stack traces if available.)

Steps to reproduce:

Have you been able to reproduce this issue with just the Firebase Unity quickstarts (this GitHub project)? No
What's the issue repro rate? (eg 100%, 1/5 etc) 100%

What happened? How can we make the problem occur?

As identified in firebase/quickstart-unity#1193, there is a problem with large sets of data with Firestore on Android, where it can use too many global refs, hitting the cap and throwing a JNI error. An initial fix was added with the 10.1.0 release which reduces the number of global refs used, but it can still potentially be hit.

Googlers see b/251869890 for more details.

[dconeybe]

FYI: If you see an error like this in the Android logcat

JNI ERROR (app bug): global reference table overflow (max=51200)

then you have hit this issue.

[dconeybe]

Also, if your stack trace includes methods like art::JavaVMExt::AddGlobalRef(...) or art::JNI::NewGlobalRef(...) then that is another strong indicator that it is this issue that you are experiencing (e.g. #586).

[dconeybe]

Update: The fix, unfortunately, is not yet ready and did not make it into last week's v10.4.0 release. We ran into some technical issues with our solution and need to do some refactoring. We hope to have the fix ready for the next release, which is scheduled for mid February 2023. If interested, you can follow the details of the fix in firebase/firebase-cpp-sdk#1176.

[dconeybe]

Update: The fix is still being finalized. We keep running into unexpected technical issues with our solution and also some competing priorities. We will update this issue when there is a build to test out.

[nitzanwilnai]

Hi, did the fix make it into 10.5.0 ?

[cherylEnkidu]

Update:

Hi, unfortunately the solution for this Global Reference exhaustion issue has been shelved for the foreseeable future.

After a 3-month investigation, the team believes that the project is a lot larger and more complex than we had originally scoped. The solution involves changes to the fundamental structure inside the SDK and without comprehensive testing we cannot prove its stability in production.

We aim to revisit this after other time-sensitive projects are complete. We apologize for the long waiting time and thank you again for your patience.

[nitzanwilnai]

I have to say this is very disappointing to hear. We have a fairly normal-sized casual game and too-many users are running out of global refs just from saving their game state. It makes Firestore a non-viable solution for us.

It would be great if you can at least find a way to clear the global ref pool manually.

[nitzanwilnai]

Hi, can you guys explain again what counts as a global reference?
So a string of 8 characters is a single reference, but an array of 8 bytes is 9 references?
We would like to re-structure our Firestore data to minimize the impact of this issue.

[ragavendranbala]

Hi @cherylEnkidu , thank you for your efforts to fix this issue. It's showing lots of crashes and ANR in play console and this affects our app listing. I can see that we using firebase remote config to dynamically update the values in many scripts and using analytics to get the info for the every button clicks. I just want to know If we reduce the remote config access to a single script and caching it will reduce the crash? Or what way I need to follow to reduce the crash? And is there's any way to inform play console to supress these errors?

[dconeybe]

Update: I have implemented a prototype for a fix for this "global refs exhaustion" issue. I still need to get it reviewed by the team and complete my testing, but so far it looks promising. I cannot provide any ETA on the fix actually being released, not in small part due to the historically-unpredictable scope creep and complexity of the fix. But I will post updates to this issue as significant milestones are achieved.

If you'd be willing to test out the fix, please try the Unity SDK build published at https://github.com/firebase/firebase-unity-sdk/actions/runs/4999199986 (download firebase_unity_sdk.zip). If this link gets garbage collected, it can be downloaded from https://github.com/firebase/firebase-unity-sdk/releases/tag/firestore_prototype_fix_0 (thanks @jonsimantov).

This test build incorporates my fix from firebase/firebase-cpp-sdk#1325. Note that this is an unofficial build made from HEAD of the https://github.com/firebase/firebase-unity-sdk and https://github.com/firebase/firebase-cpp-sdk. That being said, I expect it to be quite stable when compared to the latest official release, 11.0.0 at the time of writing.

In particular, I'm interested in two things:

1. Does this fix the crash for you?
2. Do you notice any performance degradation with the fix?

In theory, my fix should incur a performance cost; however, in my testing the performance cost was negligible, and only even noticeable when creating hundreds of thousands of objects in a tight loop. But I'm very interested to hear if there are performance issues in "real world" applications.

Thank you all for your patience as we implement a stable, reliable, and robust fix for this problematic issue.

[dconeybe]

@nitzanwilnai Did you have a chance to test out the build in the previous comment? I've only been able to test it in my contrived examples and would like to know if it does or does not fix the Firestore crash for real customer use cases.

@AnnMic Since you commented here firebase/quickstart-unity#1193 (comment), I thought I'd tag you to see if you are still experiencing the crash in Firestore. If so, would you be able to try out the test build as well and see if it improves your crash rate?

Thank you both for reporting the issue and bearing with us as we try to solve it correctly and robustly.

[dconeybe]

This bug has been fixed in the Unity SDK v11.3.0 release yesterday, July 19, 2023.

https://firebase.google.com/support/release-notes/unity#version_1130_-_july_19_2023

Please comment or open a new issue if you still see this crash or notice performance degradation from this fix.