Doesn't work with Yarn & private NPM packages on CircleCI

[adambiggs]

I've been using Yarn with serverless-webpack to deploy my functions for a while now without any issues (on OSX)... But I can't seem to get continuous deployment working on CircleCI with the same configuration.

I get the following error on CircleCI:

Serverless: Packing external modules: lodash.defaultsdeep@^4.6.0, jimp@0.2.27, @my-company/my-private-package@2.2.0
 
  Error --------------------------------------------------
 
     Command failed: npm install lodash.defaultsdeep@^4.6.0
     jimp@0.2.27 @my-company/my-private-package@2.2.0  --save
npm
     ERR! Linux 3.13.0-105-generic
npm ERR! argv "/opt/circleci/nodejs/v6.9.2/bin/node"
     "/opt/circleci/nodejs/v6.9.2/bin/npm" "install" "lodash.defaultsdeep@^4.6.0"
     "jimp@0.2.27" "@my-company/my-private-package@2.2.0" "--save"
npm
     ERR! node v6.9.2
npm ERR! npm  v3.10.9
npm ERR! code
     E404

npm ERR! 404 Not found : @my-company/my-private-package
npm
     ERR! 404 
npm ERR! 404  '@my-company/my-private-package' is
     not in the npm registry.
npm ERR! 404 You should bug
     the author to publish it (or use the name yourself!)
npm
     ERR! 404 
npm ERR! 404 Note that you can also install
     from a
npm ERR! 404 tarball, folder, http url, or git
     url.

npm ERR! Please include the following file with
     any support request:
npm ERR!     /home/ubuntu/my-serverless-service/.webpack/npm-debug.log

 
     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
 
     Please report this error. We think it might be a bug.
 
  Your Environment Information -----------------------------
     OS:                 linux
     Node Version:       6.9.2
     Serverless Version: 1.4.0

I have no idea why, but it seems like serverless-webpack depends on NPM's cache for some reason...

$ yarn install
$ sls deploy
Error! 😢

$ npm install
$ sls deploy
Success! 😕

$ npm install
$ rm -rf node_modules
$ yarn install
$ sls deploy
Success!? 🤔

$ npm install
$ rm -rf node_modules
$ npm cache clean
$ yarn install
$ sls deploy
Error!? 😵

Does this have something to do with the npm-programmatic module? Why does serverless-webpack need to install packages itself instead of just using what's already installed in node_modules?

[mhodgson]

I ran into this same problem and I think it's because npm-programmatic isn't picking up the .npmrc in the project directory. I was able to fix this by running cp .npmrc ~/.npmrc in my Circle build before trying to deploy the service. Hope this helps.

[adambiggs]

Thanks @mhodgson, I'll give that a try. But still seems weird that something like npm-programatic is needed at all.

I really want to take advantage of yarn.lock to make sure CircleCI deploys are consistent between builds... Doesn't look like there's a way to do that currently, so using Yarn outside serverless-webpack just gives the user a false sense of security.

[rosskevin]

Related: circleci/circleci-docs#593

[mhodgson]

@adambiggs Agreed. I'm wondering why we need to run npm to install the packages at all.

It seems we should just be able to copy them directly from the node_modules folder. Wouldn't that ensure deps are kept consistent? Are there risks to doing this?

Maybe we should put together a PR testing this strategy.

[adambiggs]

@mhodgson I'm guessing the thinking behind using npm-programatic was to reduce the chance of human error (e.g. user changes package.json but forgets to run $ npm install before deploying).

Maybe this makes sense as default behaviour... I guess that's debatable.

Either way:

• It should be clearly documented
• There should be some way to disable it in favour of copying dependencies directly from the project's node_modules

[edmofro]

Agree with @adambiggs, also during debug I sometimes put some console statements in a dependency, but at the moment there is no way for my debug code to run on production. Well, there are ways...but it was unexpected that my changes to a dependency in the node_modules folder didn't make it to Lambda.

[HyperBrain]

I will close this very old task for now. Thanks for the discussion. The latest versions of the plugin (V3 and V4) have some improvements in the packaging (and V4 now even offers events that can be hooked by a further plugin). So we should start a completely new discussion if needed, and check, if and how the new implementation can support deploying a copy of the local node modules (maybe only for sls deploy function which is per se a debug functionality).

Feel free to continue the discussion.

[aterreno]

Just seen this running latest versions of serverless, of this plugin too on CircleCI.
Only solution seems to create or copy an .npmrc file into ~/

[beanaroo]

We're using Artifactory and experience the same problem on GitLab. Moving the npmrc prior to deploy works, as does setting the packager to yarn in serverless.yml as:

custom:
  webpack:
    packager: 'yarn' 

This is preferable as it contains operations within the build directory.

If the defect is in an external project, do we have a linked issue? If it is limited to serverless-webpack, should we consider reopening this one?