Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to deploy Firebase app using service account default credentials #1970

Closed
miumiu0917 opened this issue Feb 14, 2020 · 19 comments · Fixed by #1973
Closed

Unable to deploy Firebase app using service account default credentials #1970

miumiu0917 opened this issue Feb 14, 2020 · 19 comments · Fixed by #1973

Comments

@miumiu0917
Copy link

[REQUIRED] Environment info

firebase-tools: 7.13.0

Platform: Ubuntu on Cloudbuild
from cloud-builders-community

[REQUIRED] Test case

  • Create any Firebase app
  • Attempt to deploy, using cloudbuld with a service account
  • Note that even if the service account is assigned the Owner role, this still fails.

[REQUIRED] Steps to reproduce

  • Attempt to deploy by running firebase deploy

[REQUIRED] Expected behavior

succesfully deployed.

[REQUIRED] Actual behavior

Starting Step #4 - "Deploy to Firebase Hosting"
Step #4 - "Deploy to Firebase Hosting": Pulling image: gcr.io/mike-242702/firebase
Step #4 - "Deploy to Firebase Hosting": Using default tag: latest
Step #4 - "Deploy to Firebase Hosting": latest: Pulling from mike-242702/firebase
Step #4 - "Deploy to Firebase Hosting": 3192219afd04: Already exists
Step #4 - "Deploy to Firebase Hosting": 17c160265e75: Already exists
Step #4 - "Deploy to Firebase Hosting": cc4fe40d0e61: Already exists
Step #4 - "Deploy to Firebase Hosting": 9d647f502a07: Already exists
Step #4 - "Deploy to Firebase Hosting": c8bd59397b26: Already exists
Step #4 - "Deploy to Firebase Hosting": f9289dfd19be: Already exists
Step #4 - "Deploy to Firebase Hosting": 9553b08b6292: Pulling fs layer
Step #4 - "Deploy to Firebase Hosting": 37581b1ce15a: Pulling fs layer
Step #4 - "Deploy to Firebase Hosting": 45ea08f1ad05: Pulling fs layer
Step #4 - "Deploy to Firebase Hosting": 134f3de54272: Pulling fs layer
Step #4 - "Deploy to Firebase Hosting": 2d088f82ff68: Pulling fs layer
Step #4 - "Deploy to Firebase Hosting": 134f3de54272: Waiting
Step #4 - "Deploy to Firebase Hosting": 2d088f82ff68: Waiting
Step #4 - "Deploy to Firebase Hosting": 45ea08f1ad05: Verifying Checksum
Step #4 - "Deploy to Firebase Hosting": 45ea08f1ad05: Download complete
Step #4 - "Deploy to Firebase Hosting": 37581b1ce15a: Verifying Checksum
Step #4 - "Deploy to Firebase Hosting": 37581b1ce15a: Download complete
Step #4 - "Deploy to Firebase Hosting": 2d088f82ff68: Verifying Checksum
Step #4 - "Deploy to Firebase Hosting": 2d088f82ff68: Download complete
Step #4 - "Deploy to Firebase Hosting": 134f3de54272: Verifying Checksum
Step #4 - "Deploy to Firebase Hosting": 134f3de54272: Download complete
Step #4 - "Deploy to Firebase Hosting": 9553b08b6292: Verifying Checksum
Step #4 - "Deploy to Firebase Hosting": 9553b08b6292: Download complete
Step #4 - "Deploy to Firebase Hosting": 9553b08b6292: Pull complete
Step #4 - "Deploy to Firebase Hosting": 37581b1ce15a: Pull complete
Step #4 - "Deploy to Firebase Hosting": 45ea08f1ad05: Pull complete
Step #4 - "Deploy to Firebase Hosting": 134f3de54272: Pull complete
Step #4 - "Deploy to Firebase Hosting": 2d088f82ff68: Pull complete
Step #4 - "Deploy to Firebase Hosting": Digest: sha256:d7764e447534464a4ae049f7fa64f55ec4433a93e4a097003ab20e1b922b3d19
Step #4 - "Deploy to Firebase Hosting": Status: Downloaded newer image for gcr.io/mike-242702/firebase:latest
Step #4 - "Deploy to Firebase Hosting": gcr.io/mike-242702/firebase:latest
Step #4 - "Deploy to Firebase Hosting": [2020-02-14T03:27:36.389Z] ----------------------------------------------------------------------
Step #4 - "Deploy to Firebase Hosting": [2020-02-14T03:27:36.392Z] Command:       /usr/local/bin/node /usr/local/bin/firebase deploy --project develop --debug
Step #4 - "Deploy to Firebase Hosting": [2020-02-14T03:27:36.393Z] CLI Version:   7.7.0
Step #4 - "Deploy to Firebase Hosting": [2020-02-14T03:27:36.393Z] Platform:      linux
Step #4 - "Deploy to Firebase Hosting": [2020-02-14T03:27:36.393Z] Node Version:  v13.8.0
Step #4 - "Deploy to Firebase Hosting": [2020-02-14T03:27:36.408Z] Time:          Fri Feb 14 2020 03:27:36 GMT+0000 (Coordinated Universal Time)
Step #4 - "Deploy to Firebase Hosting": [2020-02-14T03:27:36.408Z] ----------------------------------------------------------------------
Step #4 - "Deploy to Firebase Hosting": [2020-02-14T03:27:36.409Z] 
Step #4 - "Deploy to Firebase Hosting": [2020-02-14T03:27:36.418Z] > command requires scopes: ["email","openid","https://www.googleapis.com/auth/cloudplatformprojects.readonly","https://www.googleapis.com/auth/firebase","https://www.googleapis.com/auth/cloud-platform"]
Step #4 - "Deploy to Firebase Hosting": [2020-02-14T03:27:36.419Z] > attempting to authenticate via app default credentials
Step #4 - "Deploy to Firebase Hosting": [2020-02-14T03:27:36.525Z] TypeError: Cannot create property 'refresh_token' on string 'Not Found
Step #4 - "Deploy to Firebase Hosting": '
Step #4 - "Deploy to Firebase Hosting":     at /usr/local/lib/node_modules/firebase-tools/node_modules/google-auth-library/lib/auth/oauth2client.js:208:28
Step #4 - "Deploy to Firebase Hosting":     at /usr/local/lib/node_modules/firebase-tools/node_modules/google-auth-library/lib/auth/computeclient.js:85:7
Step #4 - "Deploy to Firebase Hosting":     at Request._callback (/usr/local/lib/node_modules/firebase-tools/node_modules/google-auth-library/lib/transporters.js:106:7)
Step #4 - "Deploy to Firebase Hosting":     at Request.self.callback (/usr/local/lib/node_modules/firebase-tools/node_modules/request/request.js:185:22)
Step #4 - "Deploy to Firebase Hosting":     at Request.emit (events.js:321:20)
Step #4 - "Deploy to Firebase Hosting":     at Request.EventEmitter.emit (domain.js:485:12)
Step #4 - "Deploy to Firebase Hosting":     at Request.<anonymous> (/usr/local/lib/node_modules/firebase-tools/node_modules/request/request.js:1154:10)
Step #4 - "Deploy to Firebase Hosting":     at Request.emit (events.js:321:20)
Step #4 - "Deploy to Firebase Hosting":     at Request.EventEmitter.emit (domain.js:485:12)
Step #4 - "Deploy to Firebase Hosting":     at IncomingMessage.<anonymous> (/usr/local/lib/node_modules/firebase-tools/node_modules/request/request.js:1076:12)
Step #4 - "Deploy to Firebase Hosting": 
Step #4 - "Deploy to Firebase Hosting": Error: An unexpected error has occurred.
Finished Step #4 - "Deploy to Firebase Hosting"
ERROR
ERROR: build step 4 "gcr.io/mike-242702/firebase" failed: step exited with non-zero status: 2
@diogoterremoto
Copy link

diogoterremoto commented Feb 14, 2020

Same thing happening today on a Cloud Build environment with the correct IAM roles to deploy to Firebase Hosting.

@miumiu0917 miumiu0917 changed the title Unable to deploy Firebase functions using service account app default credentials Unable to deploy Firebase app using service account app default credentials Feb 14, 2020
@miumiu0917 miumiu0917 changed the title Unable to deploy Firebase app using service account app default credentials Unable to deploy Firebase app using service account default credentials Feb 14, 2020
@samtstern
Copy link
Contributor

@miumiu0917 and @diogoterremoto this looks very similar to this issue:
#1966

It looks like a Cloud Build issue to me, you're getting a 404 from an OAuth endpoint which implies some sort of network issue inside the container. As you can see in the stack trace the 404 happens with google-auth-library rather than inside firebase-tools specific code.

Have you tried reporting this issue to Cloud Build directly?

@wvanderdeijl
Copy link
Contributor

wvanderdeijl commented Feb 14, 2020

We are suffering from this same issue (unable to use firebase-tools cli on google cloud build). I am still investigating but could this be causes by the deprecation of the GCP metadata endpoints? https://cloud.google.com/compute/docs/migrating-to-v1-metadata-server#supported-google-libraries

This requires GCP auth related libraries to be updated and the version of google-auto-auth that is used by firebase-tools is very old which also means it is using a very old version of gcp-metadata and gcp-auth-library:

$ npm ls gcp-metadata
├─┬ firebase-tools@7.13.0
│ └─┬ google-auto-auth@0.7.2
│   └── gcp-metadata@0.3.1
└─┬ googleapis@46.0.0
  └─┬ google-auth-library@5.8.0
    └── gcp-metadata@3.3.0
$ npm ls google-auth-library
├─┬ firebase-admin@8.9.0
│ ├─┬ @google-cloud/firestore@3.3.3
│ │ └─┬ google-gax@1.13.0
│ │   └── google-auth-library@5.8.0  deduped
│ └─┬ @google-cloud/storage@4.2.0
│   ├─┬ @google-cloud/common@2.2.5
│   │ └── google-auth-library@5.8.0  deduped
│   └─┬ gcs-resumable-upload@2.3.2
│     └── google-auth-library@5.8.0  deduped
├─┬ firebase-tools@7.13.0
│ ├─┬ @google-cloud/pubsub@1.5.0
│ │ └── google-auth-library@5.8.0  deduped
│ ├─┬ google-auto-auth@0.7.2
│ │ └── google-auth-library@0.10.0
│ └─┬ google-gax@1.12.0
│   └── google-auth-library@5.8.0  deduped
└─┬ googleapis@46.0.0
  ├── google-auth-library@5.8.0
  └─┬ googleapis-common@3.2.1
    └── google-auth-library@5.8.0  deduped

The page about the metadata server deprecation lists a number of nodejs libraries that need updating. It seems like at least gcp-metadata and google-auth-library that firebase-tools uses are too old

@samtstern
Copy link
Contributor

@wvanderdeijl I think that might be it! I will dig into it, thank you for that.

@wvanderdeijl
Copy link
Contributor

Thanks for the quick response. In the meanwhile I am trying to create my own workaround as we are currently unable to perform an important production deployment.

@samtstern
Copy link
Contributor

Working on this in #1973 ... do you think you could try installing firebase-tools from my branch and seeing it that is a workaround for you?

@wvanderdeijl
Copy link
Contributor

this only occurs on google cloud build (due to the metadata server usage). So a roundtrip to gcb for an attempt takes some time. Do you know the easiest way to install firebase-tools from your branch in a script on gcb?

@samtstern
Copy link
Contributor

You should be able to do:

npm install -g git://github.com/firebase/firebase-tools.git#ss-update-auto-auth

@wvanderdeijl
Copy link
Contributor

wvanderdeijl commented Feb 14, 2020

Installing from git works fine locally, but it fails when running in the cypress/base:10.16.0 docker container we are using in google cloud build:

$ docker run -it cypress/base:10.16.0 bash
root@ecc2be77d2d8:/# npm install -g --unsafe-perm git://github.com/firebase/firebase-tools.git#ss-update-auto-auth
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@~2.1.2 (node_modules/firebase-tools/node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.1.2: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

npm ERR! path /usr/local/lib/node_modules/firebase-tools/lib/bin/firebase.js
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall chmod
npm ERR! enoent ENOENT: no such file or directory, chmod '/usr/local/lib/node_modules/firebase-tools/lib/bin/firebase.js'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

It seems like the lib directory is not created, so creating the symlink for the firebase binary fails. If I run npm install with --bin-links=false to skip that phase I can see that the lib folder is missing:

root@c1f0fe1ecce9:/# ls -al /usr/local/lib/node_modules/firebase-tools
total 56
drwxr-xr-x   5 root root   4096 Feb 14 21:02 .
drwxrwxr-x   1 root staff  4096 Feb 14 21:02 ..
-rw-r--r--   1 root root     79 Oct 26  1985 CHANGELOG.md
-rw-r--r--   1 root root   1075 Oct 26  1985 LICENSE
-rw-r--r--   1 root root   9314 Oct 26  1985 README.md
drwxr-xr-x 412 root root  12288 Feb 14 21:02 node_modules
-rw-r--r--   1 root root   5371 Feb 14 21:02 package.json
drwxr-xr-x   2 root root   4096 Feb 14 21:02 standalone
drwxr-xr-x   5 root root   4096 Feb 14 21:02 templates

I'll keep you posted

@samtstern
Copy link
Contributor

@wvanderdeijl ah ok sorry about the issue there! You could also try this to install the firebase-tools branch on your machine:

git clone -b ss-update-auto-auth https://github.com/firebase/firebase-tools
cd firebase-tools
npm install
npm run build
npm link

@wvanderdeijl
Copy link
Contributor

I finally got it to work by not running the npm install as root as that will not completely run the prepare script. This did the trick:

su node -c "npm install"

This means I run the npm install with the non-privileged user node. This is all unrelated to this specific firebase-tools issue, so sorry for spamming this issue.

Anyhow; the npm install completed at google cloud build and I can confirm that the deployment of firebase now runs fine using your branch and the automatic auth discovery at the google cloud build server.

Should I keep this workaround in place or do you plan to release a new version with the fix very soon?

@samtstern
Copy link
Contributor

@wvanderdeijl thanks for confirming! Depends on your definition of "soon". This fix seems simple and important so we can get a new version out early next week, but we can't get it out today (we don't release on Fridays).

@wvanderdeijl
Copy link
Contributor

Thanks for the update. I am already very impressed with the very quick response times. I understand that releasing a new version on a friday as a "one man job" is not the best idea in the world.
I'll use the workaround for now and will check early next week for a new release

Thanks again!

@velith
Copy link

velith commented Feb 17, 2020

We had the same issue with our deployments and it is resolved by using branch mentioned 👍 A bit funny this "update" that made the old library not working anymore happened on Valentine's Day... where was the love? 😄 ❤️

@samtstern
Copy link
Contributor

The fix for this bug has been released in version 7.13.1

@rajivsubra1981
Copy link

Sorry for the basic q - we've been facing the same error on the deploy step on cloudbuild (last successful deploy was Feb 13th). I've updated firebase tools on my machine - but that doesnt seem to fix it on cloudbuild. How does one update the firebase tools version on cloudbuild

@crtag
Copy link

crtag commented Feb 19, 2020

@rajivsubra1981 Had the same issue, solved by rebuilding docker image for firebase community builder from here https://github.com/GoogleCloudPlatform/cloud-builders-community/tree/master/firebase, just follow the instruction in the root project readme. Once it was uploaded I also deleted the old image and everything worked fine. ,

@rajivsubra1981
Copy link

@crtag Thank you! I ran gcloud builds submit --config cloudbuild.yaml inside \cloud-builders-community\firebase
and it fixed it.

@miumiu0917
Copy link
Author

@samtstern
Thank you!!
It worked fine for me!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants