Bandwidth Exhausted while doing Firestore calls through NodeJS Server deployed on Google App Engine.

[zulqarnainhanif]

• Node.js version: v10.16.0
• npm version: 6.10.2
• google-cloud-node version: Google Cloud SDK 262.0.0

The following are the dependencies used in my NodeJs project.

"dependencies": {
    "@google-cloud/firestore": "^2.2.4",
    "@google-cloud/storage": "^3.0.2",
    "@sendgrid/mail": "^6.4.0",
    "axios": "^0.19.0",
    "body-parser": "^1.19.0",
    "express": "^4.17.1",
    "firebase-admin": "^8.2.0",
    "is-base64": "^0.1.0",
    "moment": "^2.24.0",
    "qs": "^6.8.0"
  },

While doing query request on firestore, sometime we get the error mentioned below and all the calls to firestore after that starts giving the same error. After another deployment with no changes all the apis works fine. I am still unable to figure out the leading cause of this error.

2019-09-25 12:33:27 default[20190924t162236]  { Error: Bandwidth exhausted
2019-09-25 12:33:27 default[20190924t162236]      at Http2CallStream.call.on (/srv/node_modules/@grpc/grpc-js/build/src/call.js:68:41)
2019-09-25 12:33:27 default[20190924t162236]      at Http2CallStream.emit (events.js:203:15)
2019-09-25 12:33:27 default[20190924t162236]      at process.nextTick (/srv/node_modules/@grpc/grpc-js/build/src/call-stream.js:71:22)
2019-09-25 12:33:27 default[20190924t162236]      at process._tickCallback (internal/process/next_tick.js:61:11)
2019-09-25 12:33:27 default[20190924t162236]    code: 8,
2019-09-25 12:33:27 default[20190924t162236]    details: 'Bandwidth exhausted',
2019-09-25 12:33:27 default[20190924t162236]    metadata: Metadata { options: undefined, internalRepr: Map {} } }

Can someone please explain how to resolve this issue or handle this error so that it doesn't effect the other apis?

[schmidt-sebastian]

@zulqarnainhanif Thanks for filing this issue and sorry to hear about your troubles. We have seen other reports internally and believe that the underlying issue is a RESOURCE_EXHAUSTED error from the backend. To debug your case, we will likely need project specific information, which will help us look at the workload at the time you experienced this error.

Do you mind sharing your Project ID with us via a customer service request (via https://cloud.google.com/support/)? Thank you!

I will close this issue for now as this is likely project specific.

[schmidt-sebastian]

I am re-opening this for now as @mikelehen tracked down an internal root cause for these error messages. The error seems to only reproduce with Node v10.16.3 and v12.11.0 and is likely caused by nodejs/node#29223

You can work around this by downgrading the network layer for Firestore to an implementation that does not rely on Node's HTTP/2 network stack. This can be done as such:

const admin = require("firebase-admin");
const grpc = require("grpc");

admin.initializeApp();

const firestore = admin.firestore();
firestore.settings({grpc});

[schmidt-sebastian]

@zulqarnainhanif We were able to confirm that nodejs/node#29223 is indeed the root cause. We hope to get this fixed in the underlying Node implementation soon.

[schmidt-sebastian]

The fix for the underlying issue in the HTTP2 layer will be part of Node 10.18.1 (nodejs/node#30796), Node 12.3.2 (nodejs/node#30865) and Node 12.4.1 (nodejs/node#31069). We still have to wait for these builds to be released and become available on GCP, but since there is nothing to be done in this library, I am closing this issue for now. Thank you for your patience.

[arashkay]

We are facing the same issue Bandwidth exhausted in our production environment when requests are high.

I tried your support link @schmidt-sebastian but it seems there's no place to create a ticket unless we pay for it and upgrade to paid support plans?

Can you please let us know if this is something that will be fixed soon? or what solution we should have meanwhile?

[schmidt-sebastian]

@arashkay Unfortunately, you will have to wait until Node 10.18.1 is deployed to our container frameworks. Per https://cloud.google.com/functions/docs/concepts/exec, the version is currently at 10.15.3.

[ryansydnor]

This is also happening to me in the Node 11.12 Docker image. firestore.settings({grpc}); seems to have fixed the problem for now!

[schmidt-sebastian]

@ryansydnor firestore.settings({grpc}) bypasses this problem by using a different HTTP/2 stack.

When you can switch to one of the Node version ranges mentioned above (Node 10.18.1+, Node 12.3.2+ and Node 12.4.1+), then you can switch back to the main GRPC stack. This should again reduce your binary size and startup time.

Do note that Node's version ranges are distinct and that the bug is addressed individually in each version - i.e. the fix in 10.1.8.1 is not part of Node 11.x or Node 12.0.

[asifmadnan]

If I use current stable node version 12.16.2 will it be sufficient?

[schmidt-sebastian]

It should be.

[dooleyb1]

Experiencing a similar issue when performing some reasonably light stress testing on my API which interacts with Firestore using "firebase-admin": "^8.12.0" (NodeJS).

Performing a stress test which replicates 600 users sending 4 requests each arriving at a rate of around 10 users p/s over 60s (2400 API requests in total each of which perform several reads and writes).

I am doing such a stress test as I have recently moved to Distributed Counters off the back of an issue I was having in relation to contention issues - see firebase/firebase-admin-node#856 (comment).

Approx 2k of these requests succeed as expected and around 400 fail due to the following error:

{"code":8,"details":"Bandwidth exhausted","metadata":{"internalRepr":[],"options":{}}}

[dooleyb1]

pc});

This seems to have resolved the Bandwidth exhausted issue. Thanks!

[Limule]

@dooleyb1 Hi, how did you solve it ?

[klon]

We are seeing this issue randomly occurring in our App Engine Flex NodeJS instances running Node 14.16.1 and Firestore Admin SDK 9.2.0. Once it starts all further Firestore requests fail with Error: 8 RESOURCE_EXHAUSTED: Bandwidth exhausted rendering the instance useless and forcing us to manually delete it in the App Engine console. Extremely unfortunate behavior that we need a speedy resolution to.

[navidshad]

I had the same problem for reading all data from firebase, after a while, I figured out if we stop this process and run a new process we can pass the error and continue the job.

then I used child-process and it helped me:

• I did write a parent script and a child script,
• Parent script runs child script as a child-process,
• The child goes through a collection until gets [8 RESOURCE_EXHAUSTED] error, then sends a message to the parent to inform it of the error.
• Then parent kills the child and creates a new one and tells it where to start reading again.

This is a solution that works 100 percent, but it's a little advance, and beginners-intermediates may not able to implement it.

I have written a gist and explained full implemention:
https://gist.github.com/navidshad/973e9c594a63838d1ebb8f2c2495cf87

[kirillgroshkov]

• Then parent kills child and create a new one and tells it where to start reading again.

@navidshad how do you pass the "where to start reading again" part? Do you use some kind of cursor?

I tried the same approach with child-process some years ago, it almost worked, but I think I got stuck with cursors (it was datastore, not firestore, at that moment, if it matters).

[navidshad]

• Then parent kills child and create a new one and tells it where to start reading again.

@navidshad how do you pass the "where to start reading again" part? Do you use some kind of cursor?

I tried the same approach with child-process some years ago, it almost worked, but I think I got stuck with cursors (it was datastore, not firestore, at that moment, if it matters).

Actually, I have written a cursor that manages everything relative to that issue. I can put the code here.

This is a gist I have written today to explain this solution:
https://gist.github.com/navidshad/973e9c594a63838d1ebb8f2c2495cf87