Support yarn 2 version #642

reskir · 2020-09-14T13:57:15Z

Add support for `Yarn 2.*`

Description

For bug reports:

What went wrong?
Running sls package with yarn 2.22.2 results in error:

Error: yarn install --frozen-lockfile --non-interactive failed with code 1

What did you expect should have happened?
Build package without any errors
What was the config you used?

    webpack:
        webpackConfig: ./buildWebpackConfig.js
        includeModules: true
        packager: 'yarn'
        excludeFiles: src/**/__tests__/*

What stacktrace or error message from your provider did you see?

For feature proposals:

Update this command

serverless-webpack/lib/packagers/yarn.js

Line 120 in 38135e4

const args = [ 'install', '--frozen-lockfile', '--non-interactive' ];

So it will executed succesfully with yarn 2.* version. Example: yarn install --immutable --immutable-cache
https://yarnpkg.com/cli/install

Additional Data

Serverless-Webpack Version you're using: 5.3.5
Webpack version you're using: 4.29.6
Serverless Framework Version you're using: 1.74.1
Operating System: darwin
Stack Trace (if available):

  Error: yarn install --frozen-lockfile --non-interactive failed with code 1
      at ChildProcess.<anonymous> (/Users/user/projects/lambda/node_modules/serverless-webpack/lib/utils.js:91:16)
      at ChildProcess.emit (events.js:223:5)
      at ChildProcess.EventEmitter.emit (domain.js:475:20)
      at maybeClose (internal/child_process.js:1021:16)
      at Process.ChildProcess._handle.onexit (internal/child_process.js:283:5)

The text was updated successfully, but these errors were encountered:

francisu · 2020-10-01T02:54:53Z

There is also an issue with using the "yarn list" command that happens before the install. Sadly, yarn 2 has not implemented the yarn list command. yarnpkg/berry#720

joshuanapoli · 2020-12-18T22:56:09Z

There is also an issue with using the "yarn list" command that happens before the install. Sadly, yarn 2 has not implemented the yarn list command. yarnpkg/berry#720

It looks like yarn info --name-only is a replacement for yarn list.
yarnpkg/berry#720 (comment)

nicolas-lescop · 2020-12-19T21:50:27Z

I tried your feature proposals but I have some issues

Serverless: WARNING: Could not check for peer dependencies of apollo-server-lambda
Serverless: WARNING: Could not check for peer dependencies of my-yarn-2-package

And install still fail

Error: yarn install --immutable --immutable-cache failed with code 1

theseyi · 2021-02-03T04:03:18Z

@reskir This issue is still open, in the interim have you found a workaround that works with serverless-webpack? I'm planning to patch it locally, wondering if there's a better solution?

reskir · 2021-02-03T08:25:54Z

@reskir This issue is still open, in the interim have you found a workaround that works with serverless-webpack? I'm planning to patch it locally, wondering if there's a better solution?

I have built my own plugin which perfectly fits with our monorepo project and yarn 2. These plugins are not silver bullets, they were created for specific needs, so I strongly recommend writing your own

theseyi · 2021-02-03T17:25:14Z

Thanks @reskir is your plugin something that you can share as a base to start with if it at least solves part of the problem?

reskir · 2021-02-04T17:56:17Z

@theseyi the main challenge is to gather all dependencies for the lambda function. So what we have done – separate each function to separate folders with package.json with dependencies.

Write hook for compilying assets on before:package:createDeploymentArtifacts

The function should accept two arguments – the root directory of monorepo and current (of lambda function) and our assets should be collected and transpiled recursively starting from lambda function (if those dependencies only used there) and iterating all the way back to the root directory of our monorepo.

const transformDirectory = async (
    babelOptions,
    shouldTranspile,
    srcDir,
    destDir,
    fileList = []
) => {
    // mkdirp to ensure destDir exists
    const [files] = await Promise.all([readdir(srcDir), mkdirp(destDir)]);
    const config = {
       appRoot, //lambdaRoot
       monoRoot
    };

    for (const file of files) {
        if (config.monoRoot && file === 'node_modules') {
            continue;
        }
        const srcFilePath = path.join(srcDir, file);
        const destFilePath = path.join(destDir, file);
        const fileStats = await stat(srcFilePath);
        if (fileStats.isDirectory()) {
            await transformDirectory(
                babelOptions,
                shouldTranspile,
                srcFilePath,
                destFilePath,
                fileList
            );
        } else if (shouldTranspile(srcFilePath)) {
            await babelFile(babelOptions, srcFilePath, destFilePath);
        } else {
            await copyFile(srcFilePath, destFilePath);
        }
    }
    return fileList;
};

shinnoki · 2021-05-16T15:59:09Z

I faced the same problem with Yarn berry monorepo and just passed sls package.

serverless-webpack@5.5.0
yarn@2.4.1
- berry enabled
- yarn workspaces
- nodeLinker: node-modules

I started with removing the abandoned options of yarn install.
https://github.com/serverless-heaven/serverless-webpack/blob/v5.5.0/lib/packagers/yarn.js#L122-L133
but still got error.

After that I found .webpack/dependencies and .webpack/service did not work as valid workspace projects.
Added them to workspaces in root package.json

{
  "name": "monorepo-root",
  "private": true,
  "workspaces": {
    "packages": [
      "packages/*",
      "packages/**/.webpack/*"
    ]
  },
  ...
}

and modified their package names so that they won't be duplicated.
https://github.com/serverless-heaven/serverless-webpack/blob/v5.5.0/lib/packExternalModules.js#L338
https://github.com/serverless-heaven/serverless-webpack/blob/v5.5.0/lib/packExternalModules.js#L390

Then sls package came to be finished but hoisted node_modules didn't appear in artifacts.
Fortunately, Yarn berry provides a great solution installConfig.hoistingLimits in package.json that we can apply hoist strategy for each package.

I'm wondering yarn list --depth=1 --json --production seems to by working fine 🤔

Here's my patch could be used with patch: protocol (https://yarnpkg.com/features/protocols)
https://gist.github.com/shinnoki/e66cdb69b5b0325878f4a8dee7dee064

florianbepunkt · 2021-05-18T17:47:08Z

@shinnoki Thank you. I'm on the same path (Yarn2 monorepo). Applying your patch, for me the package command still fails, if dependencies are excluded (e. g. for webpack incompatible modules) – but only, if I have multiple functions and set individual packing to be true. Having a single function works, but fails if I add a second.

Relevant parts of my config:

// webpack.config.js
// ...
externals: [
    nodeExternals(),
    nodeExternals({ modulesDir: path.resolve(__dirname, "../../node_modules") }),
  ],

package:
  individually: true
// serverless.yml webpack config
custom:
  webpack:
    webpackConfig: "./webpack.config.js"
    keepOutputDirectory: true
    packager: "yarn"
    packagerOptions:
      noFrozenLockfile: true
    includeModules:
      nodeModulesRelativeDir: "../../"
      forceExclude:

gregfletch · 2021-07-27T18:36:03Z

Any update on this issue and adding support for Yarn2 to this plugin?

zirkelc · 2021-09-22T13:36:51Z

I've found this serverless plugin https://github.com/Butterwire/serverless-plugin-monorepo to automatically create symlinks during deployment. Unfortunately, I still run into this error:

Error: yarn install --non-interactive failed with code 1

fabioDMFerreira · 2021-09-23T17:21:54Z

Was someone able to have the plugin working with yarn 2?

Getting the same error Error: yarn install --non-interactive failed with code 1.

// serverless.ts
// ...
  custom: {
    webpack: {
      packager: 'yarn',
      packagerOptions: {
        noFrozenLockfile: true,
      },
      includeModules: {
        nodeModulesRelativeDir: '../../',
        forceInclude: ['bull'],
        forceExclude: [
          'aws-sdk'
        ],
      },
    },
  },

// webpack.config.ts
// ...
 externals: [
    nodeExternals({
      allowlist: ['bull'],
      modulesDir: path.resolve(__dirname, '../../node_modules'),
    }),
    'aws-sdk',
  ],

j0k3r · 2021-09-23T18:55:57Z

Have you tried to launch yarn install --non-interactive manually to see the error?

fabioDMFerreira · 2021-09-23T19:23:48Z

Hi @j0k3r.

I've got this YN0050: The --non-interactive option is deprecated on running yarn install --non-interactive.

Any way to take out this flag?

fabioDMFerreira · 2021-09-23T20:21:17Z

Even if we removed the --non-interactive from the yarn packager it would launch the next error:

Usage Error: The nearest package directory ({{path/.webpack/dependencies}}) doesn't seem to be part of the project declared in {{project_path}}.

- If the project directory is right, it might be that you forgot to list {{path/.webpack/depencies}} as a workspace.
- If it isn't, it's likely because you have a yarn.lock or package.json file there, confusing the project root detection.

I've removed the yarn.lock from the root directory and I've added the directory as workspace (not sure if there are workspaces conflicts because the parent directory it's also a workspace), but the problem persists.

AwolDes · 2021-12-30T01:56:20Z

Bump on this - I'm running into this too with yarn version 3.1.0 & latest serverless-webpack version.

mrowles · 2022-01-22T22:01:50Z

Has anyone had any luck with this? I've tried everything, nothing seems to work or make sense.

AwolDes · 2022-01-22T22:44:27Z

@mrowles I ended up setting the version yarn uses for this project back to v1.22 to get it working 🤷‍♂️ not an ideal solution.

eugene-kim · 2022-01-26T22:59:21Z

There is also an issue with using the "yarn list" command that happens before the install. Sadly, yarn 2 has not implemented the yarn list command. yarnpkg/berry#720

@francisu were you ever able to get this working with either yarn v1 or berry?

francisu · 2022-01-27T01:48:12Z

There is also an issue with using the "yarn list" command that happens before the install. Sadly, yarn 2 has not implemented the yarn list command. yarnpkg/berry#720

@francisu were you ever able to get this working with either yarn v1 or berry?

I'm using serverless-webpack 5.5.4 with yarn 3 (berry) and it's working fine. I do specify "includeModules: false" in my serverless configuration and have my own webpack.config.

eugene-kim · 2022-01-27T01:56:24Z

There is also an issue with using the "yarn list" command that happens before the install. Sadly, yarn 2 has not implemented the yarn list command. yarnpkg/berry#720

@francisu were you ever able to get this working with either yarn v1 or berry?

I'm using serverless-webpack 5.5.4 with yarn 3 (berry) and it's working fine. I do specify "includeModules: false" in my serverless configuration and have my own webpack.config.

@francisu got it, thanks for responding. I've started to get my feet wet in the serverless world. a couple questions if you don't mind:

how are you ultimately packaging your dependencies? something custom via webpack? lambda layers?
are you avoiding the issues that some of the other commentors in this thread are running into largely because you've set includeModules to false?

many thanks

francisu · 2022-01-27T02:39:13Z

@eugene-kim

how are you ultimately packaging your dependencies? something custom via webpack? lambda layers?
Here is my webpack config:
webpack.config.js.txt

are you avoiding the issues that some of the other commentors in this thread are running into largely because you've set includeModules to false?
I think so.

I did this work a long time ago so I'm not fresh on the details. I remember hitting various problems which I think are reflected in the webpack.config.js file. I'm using version 5.66 of webpack.

eugene-kim · 2022-01-27T02:55:06Z

thanks @francisu !

eugene-kim · 2022-01-27T13:56:16Z

Even if we removed the --non-interactive from the yarn packager it would launch the next error:
Usage Error: The nearest package directory ({{path/.webpack/dependencies}}) doesn't seem to be part of the project declared in {{project_path}}.

- If the project directory is right, it might be that you forgot to list {{path/.webpack/depencies}} as a workspace.
- If it isn't, it's likely because you have a yarn.lock or package.json file there, confusing the project root detection.
I've removed the yarn.lock from the root directory and I've added the directory as workspace (not sure if there are workspaces conflicts because the parent directory it's also a workspace), but the problem persists.

@fabioDMFerreira were you ever able to get this working?

fabioDMFerreira · 2022-01-27T15:36:16Z

Yes, I had @eugene-kim. I can't remember the exact solution, but it was resolved after changing dependencies versions.

These are some packages versions we are using in our project.

serverless@2.72.0
serverless-webpack@5.6.0
webpack@5.66.0
yarn@3.1.1

mrowles · 2022-01-27T21:14:58Z

@fabioDMFerreira Same, but what are your tsconfig, webpack.config and serverless.yml > custom > webpack settings (specifically around this packing modules) - I've tried everything under the sun. I either get it compiling with no deps packed and it's tiny (throws tslib req errors in AWS) or it's huge with all deps.

fabioDMFerreira · 2022-01-27T21:29:26Z

tsconfig.json

{
  "compilerOptions": {
    "module": "commonjs",
    "declaration": true,
    "removeComments": true,
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "allowSyntheticDefaultImports": true,
    "moduleResolution": "node",
    "target": "es2017",
    "sourceMap": true,
    "baseUrl": "./src",
    "outDir": "./dist",
    "incremental": false,
    "esModuleInterop": true,
    "resolveJsonModule": true,
    "forceConsistentCasingInFileNames": true,
    "skipLibCheck": true,
  },
  "include": ["src/**/*.ts"],
  "exclude": [
    "node_modules",
    "test",
    "dist",
    "**/*spec.ts",
  ]
}

serverless.ts

...
  package: { individually: true },
  custom: {
   
      webpack: {
        packager: 'yarn',
      },
    }
 ...

webpack.config.js

const slsw = require('serverless-webpack')
const webpack = require('webpack')

module.exports = {
  entry: slsw.lib.entries,
  target: 'node',
  mode: slsw.lib.webpack.isLocal ? 'development' : 'production',
  node: {
    __dirname: true,
    __filename: true,
  },
  optimization: {
    minimize: false,
    moduleIds: 'named',
  },
  performance: { hints: false },
  externals: [
    'aws-sdk',
  ],
  devtool: 'nosources-source-map',
  module: {
    rules: [
      {
        test: /\.ts$/,
        use: 'ts-loader',
      },
    ],
  },
  resolve: {
    extensions: ['.tsx', '.ts', '.jsx', '.js'],
    symlinks: false,
  },
}

There is nothing special in my files.

@mrowles Do you have a small open source repository where you have this issue?

mrowles · 2022-01-27T21:31:09Z

@fabioDMFerreira Thanks mate, appreciate it. Your config looks pretty normal tbh + I've definitely tried this setup before. It has to be something weird I'm doing. I'll keep hacking away in the meantime.

eugene-kim · 2022-01-28T15:40:19Z

I just got packaging to work as expected for myself. Not sure if this is relevant to other situations, but I figured I'd share. I'm using yarn berry (3.1.1) and even after applying the changes that @shinnoki shared (many thanks, btw!) to remove yarn v1 cli options, packaging was an issue.

Updating my .yarnrc.yml to set

nmSelfReferences: false

Removed the self references that were messing up with webpack and causing out of memory errors.

More info on nmSelfReferences here.

sambP · 2022-04-07T15:21:17Z

Thanks @shinnoki for the patch! It is working great for a small package of the monorepo. But I'm running into errors with a larger one. The build throw a lot of warnings, which I think can not be simply ignored:

Serverless: WARNING: Could not check for peer dependencies of graphql. Set nodeModulesRelativeDir if node_modules is in different directory.

Is it something you encountered also?

.yarnrc.yml

yarnPath: ".yarn/releases/yarn-berry.cjs"
nodeLinker: node-modules
nmSelfReferences: false
nmHoistingLimits: none

root package.json

  "workspaces": [
    "packages/*",
    "packages/**/.webpack/*"
  ],

package.json

  "installConfig": {
    "hoistingLimits": "workspaces"
  },

serverless.yml

  webpack:
    webpackConfig: ./webpack.config.js
    packager: 'yarn'
    includeModules:
      nodeModulesRelativeDir: '../../'

webpack.config.js (all my projects starts with @pw/*)

  externals: [
    nodeExternals({
      allowlist: [/^@pw/],
    }),
    nodeExternals({
      allowlist: [/^@pw/],
      modulesDir: path.resolve(__dirname, '../../node_modules'),
    }),
  ],

Hideman85 · 2022-08-24T09:22:16Z

Could you please just allow an option like this 🙏

    const args = [ 'install' ];

    // Convert supported packagerOptions
    if (!packagerOptions.noNonInteractive) {
      args.push('--non-interactive');
    }

It is so annoying to have to patch the lib in postinstall 😕

pevers mentioned this issue May 31, 2021

Feature Proposal: Support monorepo structures where external dependencies are not bundled with webpack (add an example) #831

Open

eegli mentioned this issue Sep 10, 2022

chore: upgrade packages and fallback to yarn 1 eegli/spotify-history#146

Merged

jagonzalr mentioned this issue Sep 20, 2022

Add new packagerOptions: noNonInteractive to disable interactive mode when using Yarn 2 or above #1246

Merged

7 tasks

j0k3r added this to the 5.10.0 milestone Sep 20, 2022

j0k3r added enhancement packager/yarn labels Sep 20, 2022

j0k3r closed this as completed in #1246 Sep 20, 2022

j0k3r mentioned this issue Sep 26, 2022

Prepare 5.10.0 #1248

Merged

jagonzalr mentioned this issue Oct 4, 2022

Detect yarn version to dynamically not add deprecated flags #1252

Merged

7 tasks

bebbi mentioned this issue Jun 26, 2023

yarn list --depth=1 --json --production failed with code 1 #1508

Open

serverless-heaven deleted a comment from juztinlazaro Nov 14, 2023

serverless-heaven locked and limited conversation to collaborators Nov 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support yarn 2 version #642

Support yarn 2 version #642

reskir commented Sep 14, 2020 •

edited

Loading

francisu commented Oct 1, 2020

joshuanapoli commented Dec 18, 2020

nicolas-lescop commented Dec 19, 2020

theseyi commented Feb 3, 2021 •

edited

Loading

reskir commented Feb 3, 2021

theseyi commented Feb 3, 2021 •

edited

Loading

reskir commented Feb 4, 2021 •

edited

Loading

shinnoki commented May 16, 2021 •

edited

Loading

florianbepunkt commented May 18, 2021

gregfletch commented Jul 27, 2021

zirkelc commented Sep 22, 2021

fabioDMFerreira commented Sep 23, 2021

j0k3r commented Sep 23, 2021

fabioDMFerreira commented Sep 23, 2021

fabioDMFerreira commented Sep 23, 2021

AwolDes commented Dec 30, 2021

mrowles commented Jan 22, 2022

AwolDes commented Jan 22, 2022

eugene-kim commented Jan 26, 2022

francisu commented Jan 27, 2022

eugene-kim commented Jan 27, 2022

francisu commented Jan 27, 2022 •

edited

Loading

eugene-kim commented Jan 27, 2022

eugene-kim commented Jan 27, 2022

fabioDMFerreira commented Jan 27, 2022

mrowles commented Jan 27, 2022 •

edited

Loading

fabioDMFerreira commented Jan 27, 2022

mrowles commented Jan 27, 2022

eugene-kim commented Jan 28, 2022

sambP commented Apr 7, 2022 •

edited

Loading

Hideman85 commented Aug 24, 2022

Support yarn 2 version #642

Support yarn 2 version #642

Comments

reskir commented Sep 14, 2020 • edited Loading

Add support for Yarn 2.*

Description

Additional Data

francisu commented Oct 1, 2020

joshuanapoli commented Dec 18, 2020

nicolas-lescop commented Dec 19, 2020

theseyi commented Feb 3, 2021 • edited Loading

reskir commented Feb 3, 2021

theseyi commented Feb 3, 2021 • edited Loading

reskir commented Feb 4, 2021 • edited Loading

shinnoki commented May 16, 2021 • edited Loading

florianbepunkt commented May 18, 2021

gregfletch commented Jul 27, 2021

zirkelc commented Sep 22, 2021

fabioDMFerreira commented Sep 23, 2021

j0k3r commented Sep 23, 2021

fabioDMFerreira commented Sep 23, 2021

fabioDMFerreira commented Sep 23, 2021

AwolDes commented Dec 30, 2021

mrowles commented Jan 22, 2022

AwolDes commented Jan 22, 2022

eugene-kim commented Jan 26, 2022

francisu commented Jan 27, 2022

eugene-kim commented Jan 27, 2022

francisu commented Jan 27, 2022 • edited Loading

eugene-kim commented Jan 27, 2022

eugene-kim commented Jan 27, 2022

fabioDMFerreira commented Jan 27, 2022

mrowles commented Jan 27, 2022 • edited Loading

fabioDMFerreira commented Jan 27, 2022

mrowles commented Jan 27, 2022

eugene-kim commented Jan 28, 2022

sambP commented Apr 7, 2022 • edited Loading

Hideman85 commented Aug 24, 2022

reskir commented Sep 14, 2020 •

edited

Loading

Add support for `Yarn 2.*`

theseyi commented Feb 3, 2021 •

edited

Loading

theseyi commented Feb 3, 2021 •

edited

Loading

reskir commented Feb 4, 2021 •

edited

Loading

shinnoki commented May 16, 2021 •

edited

Loading

francisu commented Jan 27, 2022 •

edited

Loading

mrowles commented Jan 27, 2022 •

edited

Loading

sambP commented Apr 7, 2022 •

edited

Loading