.build/node_modules symlink vs. npm

[stesie]

Hey,

I've just recently started trying out serverless framework and this plugin just today. So sorry in case things are a bit inaccurate :)

My problem is that, together with the typescript-plugin, the core's package plugin fails to properly exclude development dependencies from the build/package. This way even a simple "hello world" function has a footprint of roughly 15MB, where it should be just about 2 kB.

Problem seems to be, that package plugin just calls npm ls --dev=true ... and npm ls --prod=true ... and diffs the output of those two. With this plugin these shell calls are done from the .build folder, yet npm ls --dev=true ... fails to properly list all the dependencies as it stumbles over the symlinking.
Seems like it doesn't expect deduped/flattened dir structure then.

Replacing both symlinkSync calls by copySync immediately fixes the problem so :)
... yet I'm unsure if there's a better way to do that ...

regards

[schickling]

Could you elaborate a bit more on the implications of replacing symlinkSync with copySync? If there are no downsides, we'd be happy to merge a PR for this.

[stesie]

So my current playground setup looks like this ...

excerpt from package.json:

{
  ...
  "devDependencies": {
    "@types/aws-lambda": "0.0.18",
    "@types/node": "^8.0.47",
    "aws-sdk": "^2.141.0",
    "serverless": "^1.23.0",
    "serverless-plugin-typescript": "^1.1.3"
  },
  "dependencies": {}
}

(so only dev dependencies, serverless installed locally)

With the replacement mentioned above in place, .build folder has a node_modules which is a real directory + package.json a copy of the file from the main folder.

Within .build folder npm ls just works as it should:

budgetier@1.0.0 /home/stesie/Projekte/budgetier/.build
├── @types/aws-lambda@0.0.18
├── @types/node@8.0.47
├─┬ aws-sdk@2.141.0
│ ├─┬ buffer@4.9.1
│ │ ├── base64-js@1.2.1
│ │ ├── ieee754@1.1.8
│ │ └── isarray@1.0.0
│ ├── crypto-browserify@1.0.9
│ ├── events@1.1.1
│ ├── jmespath@0.15.0
│ ├── querystring@0.2.0
│ ├── sax@1.2.1
│ ├─┬ url@0.10.3
│ │ ├── punycode@1.3.2
│ │ └── querystring@0.2.0 deduped
│ ├── uuid@3.1.0
│ ├─┬ xml2js@0.4.17
│ │ ├── sax@1.2.1 deduped
│ │ └── xmlbuilder@4.2.1 deduped
│ └─┬ xmlbuilder@4.2.1
│   └── lodash@4.17.4 deduped
├─┬ serverless@1.23.0
│ ├─┬ @serverless/fdk@0.5.1
│ │ ├─┬ isomorphic-fetch@2.2.1
│ │ │ ├── node-fetch@1.7.3 deduped
│ │ │ └── whatwg-fetch@2.0.3 deduped
│ │ ├── ramda@0.24.1
│ │ └─┬ url-parse@1.1.9
│ │   ├── querystringify@1.0.0
│ │   └── requires-port@1.0.0
│ ├─┬ apollo-client@1.9.3
│ │ ├── @types/graphql@0.10.2
│ │ ├─┬ apollo-link-core@0.5.4
│ │ │ ├── graphql@0.10.5 deduped
... and so on ...

yet without the change/fix in place both node_modules and package.json are symlinks to the parent's folders files. In this case npm ls fails like this:

budgetier@1.0.0 /home/stesie/Projekte/budgetier/.build
├── @serverless/fdk@0.5.1 extraneous
├── @types/aws-lambda@0.0.18
├── @types/graphql@0.10.2 extraneous
├── @types/node@8.0.47
├── agent-base@2.1.1 extraneous
├── ansi@0.3.1 extraneous
├── ansi-escapes@1.4.0 extraneous
├── ansi-regex@2.1.1 extraneous
├── ansi-styles@3.2.0 extraneous
├── apollo-client@1.9.3 extraneous
├── apollo-link-core@0.5.4 extraneous
├── archiver@1.3.0 extraneous
├── archiver-utils@1.3.0 extraneous
├── are-we-there-yet@1.1.4 extraneous
├── argparse@1.0.9 extraneous
├── array-union@1.0.2 extraneous
├── array-uniq@1.0.3 extraneous
├── async@1.5.2 extraneous
├── asynckit@0.4.0 extraneous
├─┬ aws-sdk@2.141.0
│ ├── UNMET DEPENDENCY buffer@4.9.1
│ ├── UNMET DEPENDENCY crypto-browserify@1.0.9
│ ├── UNMET DEPENDENCY events@^1.1.1
│ ├── UNMET DEPENDENCY jmespath@0.15.0
│ ├── UNMET DEPENDENCY querystring@0.2.0
│ ├── UNMET DEPENDENCY sax@1.2.1
│ ├── UNMET DEPENDENCY url@0.10.3
│ ├── uuid@3.1.0
│ ├── UNMET DEPENDENCY xml2js@0.4.17
│ └── UNMET DEPENDENCY xmlbuilder@4.2.1
├── balanced-match@1.0.0 extraneous
... again many more ...

Notice how it reports unmet dependencies with the actual dependencies + extraneous directories in node_modules folder. This seems to be due to some special handling in npm for symlinks...

Regarding the downsides of this change I think it's just about copying around lots of data instead of creating a two symlinks. Even in my simple case it's copying around 100 MBs on each build, be it invoke local or deploy. Yet with SSDs and the OS's file system cache it shouldn't hurt much. At least on my laptop it's negligible

In a way this would also "solve" issue #23 :-)

[schickling]

@pmuens could you share your view on this?

[pmuens]

Thanks for pinging @schickling 👍

Sounds like a feasible solution! Wondering if this is an edge case or smth. common 🤔.

[schickling]

Maybe a good solution is to use copy when deploying and linking while building for development?

[brettneese]

Also having issues with this.

[dls314]

Is there any update on this, or related issues (like #23, #89, #90) around using serverless-plugin-typescript on windows?

For whatever reason, the workaround that previously worked (being an administrator) isn't working for me on Windows 10 build 17134.

[markwainwright]

I'm experiencing the exact same issue described by @stesie.

I came across a workaround in a different issue that fixes this for me (#89 (comment)) – set NODE_PRESERVE_SYMLINKS=1 in your env.

I'd still love to see this get fixed though!

[RichardBradley]

I have the same issue: the "excludeDevDependencies" feature in Serverless is not working, because npm ls cannot properly read the node_modules from the symlinked dir in the .build dir.

This is not a Windows issue; npm ls will give incorrect results on a unix box when run inside the .build dir as currently created by serverless-plugin-typescript.

This largely breaks excludeDevDependencies. I am getting most of my dev dependencies included in my final zip output. It seems that NODE_PRESERVE_SYMLINKS=1 does fix this though.

I'm not sure what the best fix is from serverless-plugin-typescript's point of view is. It seems to me that the way excludeDevDependencies is coded up in zipService.js in base serverless is a bit fragile. And fixing npm itself seems a bit out of scope for serverless-plugin-typescript. Some ideas:

• Maybe serverless-plugin-typescript should just do a cp -r rather than a symlink for now? That would be a more expensive build than the current symlink approach
• Maybe serverless-plugin-typescript should explicitly copy only prod dependencies? That would complicate things here and might lead to different problems later.
• Maybe serverless-plugin-typescript can convince the dependency examination code in zipService.js to run in the "real" node_modules dir, then use the outputted exclusion paths against the .build node_modules dir later?

[RichardBradley]

I think the title of this issue should be changed to be more like "excludeDevDependencies broken by .build/node_modules symlink vs. npm". This issue is hard to find at the moment.