Firestore/triggers grpc-js No connection established errors

[clytras]

Related issues

googleapis/nodejs-firestore#768

firebase/firebase-js-sdk#1783

grpc/grpc-node#1027

Version info

node: 10.15.1

firebase-functions: 3.3.0

firebase-tools: 7.6.0

firebase-admin: 8.6.0

Test case

exports.addDriverLocation = functions.region(region).firestore
  .document('drivers/{userId}/locations/{locationId}')
  .onCreate((snap, context) => {
    const { userId = -3 } = context.params;
    const record = snap.data();
    const { location = {}} = record;
    const {
      uuid,
      coords: {
        latitude,
        longitude
      } = {},
      extras
    } = location;

    return snap.ref.set({
      userId,
      // server timestamp
      sts: admin.firestore.Timestamp.fromDate(new Date()),
      // client timestamp
      cts: admin.firestore.Timestamp.fromDate(new Date(timestamp)),
      latitude,
      longitude,
      extras
    });

Steps to reproduce

1. Create a firestore trigger function

Expected behavior

Every call should be get executed without gRPC related errors.

Actual behavior

Every now and then we get a No connection established error:

Error: No connection established
    at Http2CallStream.call.on (/srv/functions/node_modules/@grpc/grpc-js/build/src/client.js:96:45)
    at Http2CallStream.emit (events.js:194:15)
    at Http2CallStream.EventEmitter.emit (domain.js:459:23)
    at process.nextTick (/srv/functions/node_modules/@grpc/grpc-js/build/src/call-stream.js:75:22)
    at process._tickCallback (internal/process/next_tick.js:61:11)

It encountered 8600 exact errors for the past week:

For every other firestore operation, we used old grpc as described here googleapis/nodejs-firestore#768 (comment):

const Firestore = require('@google-cloud/firestore');
const grpc = require('grpc');
const firestore = new Firestore({ grpc });

but we don't know how to set old grpc for functions.region(region).firestore to get rid of all these errors.

We are not sure if here is the right place to post this issue or the https://github.com/googleapis/nodejs-firestore. If this is the case, the please close this issue and we'll go and report it on the nodejs-firestore repo.

Were you able to successfully deploy your functions?

No errors during functions deploy.

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[shankie-codes]

I'm also having this issue – just from today.

[LureOfAdventure]

I've been having the same issues as well

[mrmodelo]

Issue here as well when deploying a scheduled function. Works for about an hour then crashes. There are other threads going on but fail to address the firebase scheduled function side of things.

[tantzygames]

I've also been getting this error intermittently yesterday and today on a range of https.onCall functions

[diginikkari]

Getting also many occurencies of this error last few days with firestore triggers.

trigger: firestore create trigger.
runtime: Node.js 8
Last deployed: 16 October 2019 at 13:55:30 UTC+3
Region: europe-west1

firebase packages:
"firebase": "^7.1.0",
"firebase-admin": "8.6.0",
"firebase-functions": "^3.2.0",
"firebase-tools": "^7.5.0",

[clytras]

@mrmodelo this occurs to every kind of function that involves firestore and it's related to firestore changing grpc to grpc-js. There is a workaround for other kind of functions as you can see on my initial post, but I can't find a way to set old grpc for firestrore funtion triggers.

[adamkpurdy]

Seeing the same error on Firestore delete triggers. I just started noticing this issue this morning.

"firebase-admin": "^8.0.0",
"firebase-functions": "^3.1.0",

Region: North America / east-coast EST

[nourziady]

Same issue here, started to get No connection established error week ago

[adamkpurdy]

This looks resolved. My triggers are firing again. Can anyone else confirm?

[jhk115]

@adamkpurdy It seems to be sporadically failing on our project still.

[mrmodelo]

Since first seeing the problem, it will usually pass once after deployment, and then fail consistently after

[clytras]

@adamkpurdy since last functions deploy, I can't see any errors, but for me it's too early to tell. I'll leave this open for some days and if there won't be any more related errors I'll close this.

[adamkpurdy]

I'm seeing the error again.