Support mono-repos in deployment

[laurenzlong]

See: firebase/firebase-functions#172

Version info

3.17.3

Steps to reproduce

Expected behavior

Actual behavior

[dinvlad]

This seems to work in my setup, i.e. deploy picks up packages from the root node_modules, even though package.json for that is located under the api/ workspace (I've used a different folder name instead of functions/). Is there anything else that needs fixing here?

EDIT: Moreover, I copy package.json into api/dist to be used by deploy.

// firebase.json
  ...
  "functions": {
    "source": "api/dist"
  },
  ...

So, 2 levels of nesting still resolve the root node_modules successfully.

[orouz]

@dinvlad could you share a repo?

[dinvlad]

@orouz unfortunately not yet, it's closed source for now.

[audkar]

Does anyone managed to tackle this problem? Sharing simple example project would be very useful.

[jthegedus]

@audkar Currently I just use lerna.js.org and it's run command to execute an npm script in each subfolder with this folder structure:

- service1/
|  - .firebaserc
|  - firebase.json
- service2/
|  - .firebaserc
|  - firebase.json
- app1/
|  - .firebaserc
|  - firebase.json
- app2/
|  - .firebaserc
|  - firebase.json
- firestore/
|  - firestore.rules
|  - firestore.indexes.json
- etc...

Ensuring the firebase.json files for each service don't stomp on one another is left up to the user. Simple conventions of using function groups and multi-site name targeting means this is solved for Cloud Functions and Hosting. Still haven't got a solution for Firestore/GCS rules yet, though splitting them up may not be ideal...

discussed here previously - #1116

[audkar]

@jthegedus thank you for your reply. But I think issue of this ticket is different. I am trying to use yarn workspaces. And seems that firebase tools doesn't pickup symlink dependencies when uploading functions

[jthegedus]

Ah fair enough, I've avoided that rabbit hole myself

[dinvlad]

Could you elaborate what the issue is? As mentioned above, I just use bare Yarn with api and app workspaces in it, and I build them using yarn workspace api build && yarn workspace app build (with build script specific to each workspace). The build scripts

1. compile TS code with outDir into api/dist and app/dist respectively
2. copy the corresponding package.json files into dist directories
3. copy yarn.lock from the root folder, into dist directories

Then I just run yarn firebase deploy from the root folder, and it picks up both api/dist and app/dist without any hiccups. My firebase.json looks like

  "functions": {
    "source": "api/dist"
  },
  "hosting": {
    "public": "app/dist",

Unfortunately, I still can’t share the full code, but this setup is all that matters, afaik.

[dinvlad]

Also, I might be wrong but I think the firebase deploy script doesn’t actually use your node_modules directory. I think it just picks up the code, package.json, and yarn.lock from the dist directories, and does the rest.

[samtstern]

That's true. The default value of "functions.ignore" in firebase.json is ["node_modules"] so it's not uploaded. I believe you can override that though if you want to ship up some local modules.

…

On Mon, Jun 17, 2019, 6:58 PM Denis Loginov ***@***.***> wrote: Also, I might be wrong but I think the firebase deploy script doesn’t actually use your node_modules directory. I think it just picks up the cod, package.json, and yarn.lock from the dist directories, and does the rest. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#653?email_source=notifications&email_token=ACATB2U73VS2KIILUVRFFB3P3A6NPA5CNFSM4EOR24GKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX46ABQ#issuecomment-502915078>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACATB2U3Q2TLVBICRJ3B5OLP3A6NPANCNFSM4EOR24GA> .

[jthegedus]

@dinvlad Yes, it requires the package.json and whichever lock file you use as it installs deps in the cloud post deployment.

I believe the scenario originally outlined in the other issue was using a shared package within the workspace and some issues with scope-hoisting. As I was not using yarn this way I can only speculate from what I have read there.

[dinvlad]

@samtstern @jthegedus thanks, good to know!

[audkar]

Seems we all talk about different problems. I will try to describe yarn workspaces problem.

Problematic project

project layout

- utilities/
|  - package.json
- functions/
|  - package.json
- package.json

./package.json

{
  "private": true,
  "workspaces": ["functions", "utilities"]
}

functions/package.json

{
  <...>
  "dependencies": {
    "utilities": "1.0.0",
    <...>
  }
}

Problem

Error during function deployment:

Deployment error.
Build failed: {"error": {"canonicalCode": "INVALID_ARGUMENT", "errorMessage": "`gen_package_lock` had stderr output:\nnpm WARN deprecated left-pad@1.3.0: use String.prototype.padStart()\nnpm ERR! code E404\nnpm ERR! 404 Not Found: utilities@1.0.0\n\nnpm ERR! A complete log of this run can be found in:\nnpm ERR!     /builder/home/.npm/_logs/2019-06-18T07_10_42_472Z-debug.log\n\nerror: `gen_package_lock` returned code: 1", "errorType": "InternalError", "errorId": "1971BEF9"}}

Functions works fine locally on emulator.

Solutions tried

Uploading node_modules (using functions.ignore in firebase.json). Result is same.

My guess that it is because utilities is created as syslink in node-modules node_modules/utilities -> ../../utilities

Could it be that firebase-tools doesn't include content of symlink'ed modules when uploading (no dereferencing)?