Getting RESOURCE_EXHAUSTED randomly

[gilamran]

Operating System

macOS 13.6 (22G120)

Browser Version

Node 18 or Latest Chrome

Firebase SDK Version

10.4.0

Firebase SDK Product:

Firestore

Describe your project's tooling

nodejs or browser with firestore

Describe the problem

Using the simplest project with only one collection with 30,000 items in the collection. each doc in the collection looks like this: { id: 1 }. I'm getting RESOURCE_EXHAUSTED randomly.
The project is in Blaze!

The projectId is test2-82c47

Steps and code to reproduce issue

The simplest example will be to just paste this code into index.js and run node index.js
The error happens randomly, for me it happened around 50% of tires, so if it did not happen re-run.

This is the error that I got:

[2023-09-27T10:31:41.763Z]  @firebase/firestore: Firestore (10.4.0): GrpcConnection RPC 'Listen' stream 0x6813ca3b error. Code: 8 Message: 8 RESOURCE_EXHAUSTED: Resource has been exhausted (e.g. check quota).
[2023-09-27T10:31:41.763Z]  @firebase/firestore: Firestore (10.4.0): FirebaseError: [code=resource-exhausted]: 8 RESOURCE_EXHAUSTED: Resource has been exhausted (e.g. check quota).
[2023-09-27T10:31:41.763Z]  @firebase/firestore: Firestore (10.4.0): Using maximum backoff delay to prevent overloading the backend.

const firebase = require("firebase/compat/app");
require("firebase/compat/firestore");
require("firebase/compat/auth");

const FIREBASE_CONFIG = {
.
.
.
};

const firebaseApp = firebase.initializeApp(FIREBASE_CONFIG);
const firestore = firebaseApp.firestore();

const getData = async () => {
  console.log("starting query");
  // collection has 30,000 items, each with one object that looks like this: { id: 1 }
  const query = await firestore.collection("supportedAssets").get();
  console.log(`query returned ${query.docs.length} items`);
  firebaseApp.delete();
};

getData();

[gilamran]

Note that if I'm using firebase-admin library it does not happen!

[wu-hui]

Hi @gilamran

Thanks for the report.

We are looking into this issue, but I want to describe what is happening:

This is essentially an issue caused by slow SDK documents consumption. When a large number of documents are being returned from Firestore's backend, and the SDK is taking too long to process the incoming responses, the backend need to keep the documents in its buffer. When that buffer is fully filled, the backend has to drop the result and return a RESOURCE_EXHAUSTED error, in order for its own survival.

Of all Firestore SDKs, the Web SDK is the slowest, and thus most susceptible to this issue. This explained why the error is random (sometimes it is faster enough to consume those documents); also why firebase-admin works (it is in general faster, although this could still happen if the machine is under heavy load).

We will try to return a more meaningful error message to reduce confusion, at the very least. We are still investigating to see if we can improve the situation. I will post updates here when we have them.

In the meantime, the best you can do is to break up your large query into smaller ones, you can use pagination, or apply additional filters.

Thanks!

[gilamran]

Please take these points into account:

• The client is doing nothing with the result, no processing at all.
• The CPU is totally free, if there's some processing internally in the SDK, it's not affecting my CPU. I added X6 throttling on my CPU and it did not affect the randomness.
• The client SDK can fail on 10,000 documents and not fail on 50,000... where the admin SDK never fails...
• This issue started about a week ago, before that it worked on even larger collections with a lot more information and much more load on the CPU...

[wu-hui]

Thanks for more information.

Was there any change on your side a week ago? Did you upgrade to a newer version of the SDK for example?

This limitation always exists, but we have not eliminated the possibility that it is some change to our backend that made this worse than before. Any information that could help us isolate the likely causes is helpful.

--

On CPU load, I think that is one of the likely places where SDK is struggling to consume, it could be somehow the network load is too much for example. And also this has nothing to do with if client/SDK is actually processing the documents, the SDK was simply not fast enough in receiving (and acknowledging) the documents for some reason. Having said this, it is also possible it's some backend component that is slowing responses down, we will check those too.

[ChiragMoradiya]

I agree with @gilamran that this issue is not caused due to CPU, Memory or Network starvation on the Client (Web Browser), which is causing the delay in the processing.

Whenever we encountered firebase/firebase-android-sdk#3070 we observed everything in detail. So, I am sure about this.

[gilamran]

We changed nothing a week ago, and did not upgrade the SDK version.

[gilamran]

ok, we've contacted Google, they investigated the issue, found it, fixed it, and deployed it.
case closed.

[wu-hui]

Thanks @gilamran

To add more context, the R_E error limit still exists, and my comment #7655 (comment) still applies: if the client is too slow receiving responses backend might have to give up buffering and send this error to clients.

However, this was not what happened with this particular report. The clients in these reports were fine, there was something going on in the backend that was causing this.

Hope this would help people in the future running into this error troubleshoot.

[tiposgmbh]

Thanks for more information.
Was there any change on your side a week ago? Did you upgrade to a newer version of the SDK for example?

This limitation always exists, but we have not eliminated the possibility that it is some change to our backend that made this worse than before. Any information that could help us isolate the likely causes is helpful.

[...] Having said this, it is also possible it's some backend component that is slowing responses down, we will check those too.

We experience this problem as well. It started about a month ago. There was no change in code or SDK version.
The data grows gradually. But before, the error apparently never occurred and then it suddenly started occurring quite frequently.
It is still not consistently reproducable, however. Answers to the following questions could help us:

• Is the limit per database or finer like per connection/session/client/etc. or even finer as per collection?
• Is the limit (before the buffer is considered exhausted) a limit on number of documents or a limit on bytes?
• Is there possibly (also) any connection to caching?
  We seemed to observe a device-dependence and a fix for the error when applying CTRL+F5 in Chrome (even in Incognito).

Thank you for any help!

[wu-hui]

• Is the limit per database or finer like per connection/session/client/etc. or even finer as per collection?

It's a per client limit.

• Is the limit (before the buffer is considered exhausted) a limit on number of documents or a limit on bytes?

It's on the number of documents for now, we could change this at sometime without announcements however.

• Is there possibly (also) any connection to caching?
  We seemed to observe a device-dependence and a fix for the error when applying CTRL+F5 in Chrome (even in Incognito).

Not sure I understand your question. It could be device-dependent because different clients consume documents at different rates. However, I just want to stress that this particular reporting is induced by a backend change, and it has now been fixed. Different client consume rate could affect how soon you will reach the limit still though.

[tiposgmbh]

Thanks your for your help. It is very much appreciated.

It's on the number of documents for now, we could change this at sometime without announcements however.

• Can this limit somehow be quantified? Is there a number of documents you can tell me?
• Is this limit something new or has it been lowered recently?

We are getting this problem, by the way, using .onSnapshot(...) and at times (for hours or even days) the error doesn't occur at all and sometimes it occurs nearly everytime, doing the same exact thing with (basically) the same data every time.

One observation that is very suspicious is the fact that every time the error occurs it basically does so immediately (clearly within one second) while when it doesn't the data loads for several seconds (it's loading data on the scale of 10k to 100k documents). There seems to be these two different kinds of scenarios and no continuum between them. That seems odd given the explanation you provided. Or can you explain that?

Thank you very much!

[wu-hui]

I am not allowed to disclose the limit unfortunately, as this is not a guarantee from Firestore, and will change over time.

Are you still experiencing this now? The backend change should fixed the issue we identified. So you might be seeing a different issue, if you are still seeing it.

I also do not have an explanation for what you observed. I will forward this to our backend engineers.

[tiposgmbh]

Thanks!

Yes, we are experiencing this. But it cannot be consistently reproduced at all. Right now I'm trying to do it with network throttling in Chrome and still I can't deliberately reproduce it. It is very weird. With "Fast 3G" it took forever to load the data. But it still worked.

I was able to reproduce the error in the past without browser as well - just a node script that subscribes to two collections (with ~100k and ~10k documents data) using .onSnapshot(...) and does nothing with the data when it arrives and nothing else either.