Firestore emulator fails to retrieve refresh_token in CI, errors out

[joelpoloney]

[REQUIRED] Environment info

firebase-tools: 7.12.1

Platform: Linux

[REQUIRED] Test case

Starting the simulator in Google Cloud Build fails randomly and is causing our tests to be flaky

[REQUIRED] Steps to reproduce

Right now I'm running lerna with jest. The command is run on Node 10.15.1 on Cloud Build. Here's the yarn command I'm running: firebase emulators:exec --only firestore 'yarn jest'".

The yarn jest command looks like: "jest": "FIRESTORE_EMULATOR_HOST=localhost:9090 jest --runInBand --coverage"

When run with the --debug flag, I get some more information and a stack trace of where things blew up:

Already have image: build-image
yarn run v1.13.0
$ lerna run test --stream
lerna notice cli v3.20.2
lerna info versioning independent
lerna info Executing command in 2 packages: "yarn run test"
siteline-server: $ firebase emulators:exec --only firestore --debug 'yarn jest'
siteline-server: [2020-02-10T21:44:12.408Z] ----------------------------------------------------------------------
siteline-server: [2020-02-10T21:44:12.411Z] Command:       /usr/local/bin/node /workspace/node_modules/.bin/firebase emulators:exec --only firestore --debug yarn jest
siteline-server: [2020-02-10T21:44:12.412Z] CLI Version:   7.12.1
siteline-server: [2020-02-10T21:44:12.412Z] Platform:      linux
siteline-server: [2020-02-10T21:44:12.412Z] Node Version:  v10.15.1
siteline-server: [2020-02-10T21:44:12.412Z] Time:          Mon Feb 10 2020 21:44:12 GMT+0000 (Coordinated Universal Time)
siteline-server: [2020-02-10T21:44:12.414Z] ----------------------------------------------------------------------
siteline-server: [2020-02-10T21:44:12.414Z] 
siteline-server: [2020-02-10T21:44:12.425Z] > command requires scopes: ["email","openid","https://www.googleapis.com/auth/cloudplatformprojects.readonly","https://www.googleapis.com/auth/firebase","https://www.googleapis.com/auth/cloud-platform"]
siteline-server: [2020-02-10T21:44:12.426Z] > attempting to authenticate via app default credentials
siteline-server: [2020-02-10T21:44:12.551Z] TypeError: Cannot create property 'refresh_token' on string 'Not Found
siteline-server: '
siteline-server:     at /workspace/node_modules/google-auto-auth/node_modules/google-auth-library/lib/auth/oauth2client.js:208:28
siteline-server:     at /workspace/node_modules/google-auto-auth/node_modules/google-auth-library/lib/auth/computeclient.js:85:7
siteline-server:     at Request._callback (/workspace/node_modules/google-auto-auth/node_modules/google-auth-library/lib/transporters.js:106:7)
siteline-server:     at Request.self.callback (/workspace/node_modules/request/request.js:185:22)
siteline-server:     at Request.emit (events.js:189:13)
siteline-server:     at Request.EventEmitter.emit (domain.js:441:20)
siteline-server:     at Request.<anonymous> (/workspace/node_modules/request/request.js:1161:10)
siteline-server:     at Request.emit (events.js:189:13)
siteline-server:     at Request.EventEmitter.emit (domain.js:441:20)
siteline-server:     at IncomingMessage.<anonymous> (/workspace/node_modules/request/request.js:1083:12)
siteline-server:     at Object.onceWrapper (events.js:277:13)
siteline-server:     at IncomingMessage.emit (events.js:194:15)
siteline-server:     at IncomingMessage.EventEmitter.emit (domain.js:441:20)
siteline-server:     at endReadableNT (_stream_readable.js:1103:12)
siteline-server:     at process._tickCallback (internal/process/next_tick.js:63:19)
siteline-server: Error: An unexpected error has occurred.
siteline-server: error Command failed with exit code 2.
siteline-server: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
lerna ERR! yarn run test exited 1 in 'siteline-server'
lerna WARN complete Waiting for 1 child process to exit. CTRL-C to exit immediately.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[REQUIRED] Expected behavior

Does not crash

[REQUIRED] Actual behavior

Crashes

[samtstern]

A wild @joelpoloney appears! 👋

I thought I'd seen all the auth errors but this is a new one for me. Have you ever seen this locally or only on Cloud Build? Also are you trying to access any production resources from the emulator or should it all be self-contained?

[joelpoloney]

Hi!! 👋

We have not run into this locally... it's only appearing on Cloud Build. We're not trying to access any production resources at all -- we want to run our unit tests without talking to production. We were using the firebase-mock package before, but that has some serious memory and performance issues with it that were crippling our machines (heap sizes of 1.5gb just running tests!!) and dragging out build times. We switched to the emulator, which solved all these problems except now it's running into random auth errors.

That said, should we be auth'ing the emulator somehow? Perhaps we just set it up wrong?

[joelpoloney]

That said, we were in the middle of upgrading all of our dependencies and realized Cloud Build was still on Node 10.15. I've updated to Node 12.15.0 and Yarn 1.21.0 and will continue to monitor... but reading the error, this feels platform (node/yarn) independent.

[samtstern]

So fundamentally it looks like you're getting a 404 from some auth service, which is either a bug in Cloud Run's networking or in the google-auto-auth library.

However I think we can work around it. Here's where you're running into issues:
https://github.com/firebase/firebase-tools/blob/master/src/requireAuth.js#L18-L52

So I think what we want to do is have you authenticate with a token. If you run this on your personal machine:

$ firebase login:ci

You will get a permanent token that you can use to authorize the CI. You can then pass that to the Firebase CLI in two ways:

• Set the FIREBASE_TOKEN env var globally
• Pass it to each command with --token

As for getting it securely into Cloud Build, you can encrypt it using Cloud KMS and then get Cloud Build to bring it in for you as secretEnv. Here's an example where I did that:
https://github.com/firebase/firebaseopensource.com/blob/master/static/cloudbuild.yaml#L2

Another option is to authenticate with a service account JSON key (GOOGLE_APPLICATION_CREDENTIALS) but let's try that last.

[joelpoloney]

Ah ok great. This is probably the issue -- having a permanent token looks like it fixes it. I'll try the route with KMS and report back how it goes. Thank you for the super detailed info!

[joelpoloney]

I followed these instructions to get a variable setup: https://cloud.google.com/cloud-build/docs/securing-builds/use-encrypted-secrets-credentials#example_build_request_using_an_encrypted_variable.

That combined with your cloudbuild.yaml suggestions, I was able to get it working (on first try!). For those curious, I simplified this to just decode to the FIREBASE_TOKEN env variable directly with:

secrets:
- kmsKeyName: projects/[PROJECT-ID]/locations/global/keyRings/[RING-ID]/cryptoKeys/FIREBASE_TOKEN
  secretEnv:
    FIREBASE_TOKEN: <base64 encoded value>
steps:
- name: 'build-image'
  entrypoint: yarn
  args: ['test']
  secretEnv:
    - FIREBASE_TOKEN

[joelpoloney]

Resolution above. Closing this out. Thank you!!!

[samtstern]

@joelpoloney glad you figured it out!

@yuchenshi @markarndt maybe a good addition to Emulator + CI docs.

[samtstern]

@joelpoloney turns out there was an issue on our side too, #1973 fixes it and will be included in a patch release very soon. So you should be able to remove FIREBASE_TOKEN and move back to depending on the Cloud Build default credentials (if you want to)

[joelpoloney]

Ah good to know. Thank you for the heads up!