Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document how to bundle node externals when using Yarn Workspaces #494

Closed
Tracked by #495
jormal opened this issue Apr 23, 2019 · 13 comments
Closed
Tracked by #495

Document how to bundle node externals when using Yarn Workspaces #494

jormal opened this issue Apr 23, 2019 · 13 comments

Comments

@jormal
Copy link

jormal commented Apr 23, 2019

This is a Bug Report

Description

For bug reports:

  • What went wrong?
    When I was using only one package, web3 could be bundled successfully.
    But, I needed to restruct my project to using yarn workspace. So, I only restructed (not code edited) my project and I came across some bundling errors.
(skip warning)

ERROR in D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/scrypt/index.js
Module not found: Error: Can't resolve './build/Release/scrypt' in 'D:\Users\jormal\WebstormProjects\TX_COLLECTOR\tx-collector\node_modules\scrypt'
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/scrypt/index.js 3:19-56
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/scrypt.js/node.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/web3-eth-accounts/dist/web3-eth-accounts.esm.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/web3-eth/dist/web3-eth.esm.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/web3/dist/web3.esm.js
 @ ./dev/src/block-collector/main.js

ERROR in D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/swarm-js/node_modules/got/index.js
Module not found: Error: Can't resolve 'electron' in 'D:\Users\jormal\WebstormProjects\TX_COLLECTOR\tx-collector\node_modules\swarm-js\node_modules\got'
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/swarm-js/node_modules/got/index.js 45:20-39
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/swarm-js/lib/files.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/swarm-js/lib/api-node.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/web3-bzz/dist/web3-bzz.esm.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/web3/dist/web3.esm.js
 @ ./dev/src/block-collector/main.js

This error seems like same error in #224. But, that error of this issue is not solved..
I think it depend with #438, serverless-webpack couldn't follow the mono-repo structure.

  • What did you expect should have happened?
    I want to know the way to fixing it temporarily or supporting yarn workspace permanently.

  • What was the config you used?

const slsw = require("serverless-webpack");
const nodeExternals = require("webpack-node-externals");
require("@babel/polyfill");

module.exports = {
  mode: "development",
  entry: slsw.lib.entries,
  target: "node",
  externals: [nodeExternals()],
  module: {
    rules: [
      {
        test: /\.js$/,
        exclude: /(node_modules|bower_components)/,
        use: {
          loader: "babel-loader",
          options: {
            presets: ["@babel/preset-env"],
            plugins: [
              "@babel/plugin-transform-async-to-generator",
              "@babel/plugin-transform-regenerator"
            ]
          }
        }
      }
    ]
  }
};

.bablerc

{
  "presets": [
    ["@babel/preset-env", {
      "targets": {
        "node": "current"
      },
      "useBuiltIns": "entry"
    }]
  ],
  "plugins": [
    "@babel/plugin-transform-async-to-generator",
    "@babel/plugin-transform-regenerator"
  ]
}
  • What stacktrace or error message from your provider did you see?
    It caused at Serverless: Bundling with Webpack...
    And make message Webpack compilation error, see above
    And also, SLS_DEBUG=* makes no difference.
    Skipped warnings are
WARNING in D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/BufferUtil.js
Module not found: Error: Can't resolve '../build/Release/bufferutil' in 'D:\Users\jormal\WebstormProjects\TX_COLLECTOR\tx-collector\node_modules\websocket\lib'
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/BufferUtil.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/WebSocketFrame.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/WebSocketConnection.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/websocket.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/index.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/web3-providers/dist/web3-providers.esm.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/web3/dist/web3.esm.js
 @ ./dev/src/block-collector/main.js

WARNING in D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/Validation.js
Module not found: Error: Can't resolve '../build/Release/validation' in 'D:\Users\jormal\WebstormProjects\TX_COLLECTOR\tx-collector\node_modules\websocket\lib'
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/Validation.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/WebSocketConnection.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/websocket.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/index.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/web3-providers/dist/web3-providers.esm.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/web3/dist/web3.esm.js
 @ ./dev/src/block-collector/main.js

WARNING in D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/BufferUtil.js
Module not found: Error: Can't resolve '../build/default/bufferutil' in 'D:\Users\jormal\WebstormProjects\TX_COLLECTOR\tx-collector\node_modules\websocket\lib'
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/BufferUtil.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/WebSocketFrame.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/WebSocketConnection.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/websocket.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/index.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/web3-providers/dist/web3-providers.esm.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/web3/dist/web3.esm.js
 @ ./dev/src/block-collector/main.js

WARNING in D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/Validation.js
Module not found: Error: Can't resolve '../build/default/validation' in 'D:\Users\jormal\WebstormProjects\TX_COLLECTOR\tx-collector\node_modules\websocket\lib'
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/Validation.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/WebSocketConnection.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/websocket.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/index.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/web3-providers/dist/web3-providers.esm.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/web3/dist/web3.esm.js
 @ ./dev/src/block-collector/main.js

Similar or dependent issue(s):

Additional Data

  • Serverless-Webpack Version you're using: 5.2.0
  • Webpack version you're using:
  • Serverless Framework Version you're using: 1.40.0
  • Operating System: Windows 10 and Amazon Linux 2017.03 (both got same errors)
  • Stack Trace (if available):
@jormal
Copy link
Author

jormal commented Apr 23, 2019

Holy moly I solved it..
In, webpack.config.js

...
  externals: [nodeExternals({
    modulesDir: path.resolve(__dirname, '../../node_modules'),
  })],
...

It makes solve all of errors and warnings.

I was struggled in this issue almost whole three days... Anyway, I will close it.

@hassankhan
Copy link
Contributor

Hi @jormal, thanks for opening the issue, sorry for the late response, glad you managed to find a workaround. I think this might be worth adding to the documentation, so I'll re-open the issue with a new title.

@hassankhan hassankhan reopened this Apr 23, 2019
@hassankhan hassankhan changed the title How to bundle the web3 in using yarn workspace? How to bundle node externals when using Yarn Workspaces? Apr 23, 2019
@hassankhan hassankhan changed the title How to bundle node externals when using Yarn Workspaces? Document how to bundle node externals when using Yarn Workspaces Apr 23, 2019
@jormal
Copy link
Author

jormal commented Apr 24, 2019

(EDITED)
Hi @hassankhan, thanks for reply to me. I'm glad to my issue help your documentation.

I solved it by fixing config. But, I have tiny another issue. I think it should be announced to you.
I found some version warning in fetching dependency.
In custom.webpack of my serverless.yml

  webpack:
    webpackConfig: ./webpack.config.js
    packager: yarn
    includeModules:
      packagePath: "${self:custom.relativePathToRoot}/package.json"

After bundling, in fetching dependency, caused another warnings about disable of determining version.
Like this,

Serverless: WARNING: Could not determine version of module web3
Serverless: WARNING: Could not determine version of module web3

Web3 has dependency on package, not root. So, It was excepted in "root/package.json", on the other hands, it was contained in "root/packages/eth/package.json"
I expected packagePath in custom.webpack.includeModules can has array of path strings, but it did not.

It doesn't look like affecting anything in operation right now. But, I worried about it would be cause of trouble in maintenance.
If you have any idea about it, please notify to me.
Thank you. :)

@bradennapier
Copy link

bradennapier commented Jun 19, 2019

This actually is a problem and you can't support workspaces. This is due to the fact you bypass node externals rather than listening to it.

We need to be able to provide an array of packagePath to look at for workspaces to work properly otherwise further handling of the yarn workspace resolution would be needed to detect this automatically.

Ideally you would just follow node externals - not sure why this is custom.

Another option is to look for the workspaces property in the package.json and be sure to scan all directories for their package.json and exclude all that are present.

Either way not sure why defining the proper externals in my webpack config is not actually making those external unless i then also do node externals in the serverless-webpack config

node-externals will look in the actual node_modules follder and exclude all modules present inside. So when I want to exclude any packages in multiple packages in the workspaces that may appear:

externals: [
    'aws-sdk',
    nodeExternals({
      modulesDir: path.resolve(path.dirname(rootDir), 'shared'),
    }),
    nodeExternals({
      modulesDir: path.dirname(rootDir),
    }),
    nodeExternals({
      modulesDir: rootDir,
    }),
  ],

but i cant provide all 3 paths again to serverless-webpack. It would be better to just follow the externals property and use that.

This is required because of the following config which allows defining other module directories

resolve: {
    modules: [
      path.resolve(`${rootDir}/../shared`),
      path.resolve(`${rootDir}/../node_modules`),
      'node_modules',
    ],
  },

@danrivett
Copy link

danrivett commented Jun 2, 2020

Any update on this, I'm new to Yarn Workspaces and serverless-webpack is complaining with an error such as this when I run sls package:

yarn install --frozen-lockfile --non-interactive failed with code 1
  error An unexpected error occurred: "https://registry.yarnpkg.com/<internal package>: Not found".

I tried the above option (updating to hit my top level node_modules directory):

externals: [
nodeExternals({
  // Exclude top-level node_modules as using Yarn Workspaces as dependencies are hoisted
  // See: https://github.com/serverless-heaven/serverless-webpack/issues/494#issuecomment-485684097
  modulesDir: path.resolve(__dirname, '../../../node_modules'),
}),
],

But still no luck. Just curious if there is anything obvious to allow serverless-webpack to work with local modules accessible through Yarn Workspaces?

@Tybot204
Copy link

Tybot204 commented Jul 4, 2020

I have also been running into these same issues the past few days. I think I've solved the internal package not found issue but whitelisting it in nodeExternals:

externals: [nodeExternals({ whitelist: ["internal-package"] })]

This ensures webpack pulls the local version of your internal package.

However, I have not found a good workaround for WARNING: Could not determine version of module <package>. Webpack tries to install dependencies excluded from nodeExternals, and fails because the empty version of these packages conflicts with Yarn's lockfile. I played around with forcing Yarn to install the packages anyways (by removing --frozen-lockfile from the dependency command) but this doesn't really solve the issue. My packages build, but they don't necessarily use the correct versions since Webpack didn't know which version to install.

Has anyone found a workaround or poked into supporting yarn workspaces? I'm thinking about ditching workspaces in the near future as I've spent too much time running in circles looking for workarounds 😅

@iam-yan
Copy link

iam-yan commented Aug 12, 2020

@Tybot204 Same here..... so frustrated to work with workspaces... keep hoping for library X to support it... while now yarn is promoting pnp 😅...

What tool set would you go for if you ditched workspace? nx? lerna?

@cainrus
Copy link

cainrus commented Sep 13, 2020

@juuyan Yarn workspaces are not compatible with pnp mode.

tingc10 referenced this issue in tingc10/serverless-white-elephant Dec 10, 2020
… doesnt work well with yarn workspaces. Serverless reports module not found. This is due to a workaround to get serverless-webpack to work, but the node externals does not include the module https://github.com/serverless-heaven/serverless-webpack/issues/494\#issuecomment-637244350
@deathemperor
Copy link

Holy moly I solved it..
In, webpack.config.js

...
  externals: [nodeExternals({
    modulesDir: path.resolve(__dirname, '../../node_modules'),
  })],
...

It makes solve all of errors and warnings.

I was struggled in this issue almost whole three days... Anyway, I will close it.

you sir, save my day!

@joebernard
Copy link

Here is a Webpack config that works for my serverless app that uses Yarn Workspaces. This app deploys to AWS. It is far from perfect but it does work. Hope it helps someone.

  mode: slsw.lib.webpack.isLocal ? "development" : "production",
  entry: slsw.lib.entries,
  target: "node",
  externals: [
    {
      "aws-sdk": "commonjs aws-sdk",
      bufferutil: "commonjs bufferutil",
      "utf-8-validate": "commonjs utf-8-validate",
    },
  ],
  plugins: [
    new ProgressBarPlugin(),
    new webpack.IgnorePlugin(/^\.\/locale$/, /moment$/),
    new webpack.DefinePlugin({
      "process.env.FLUENTFFMPEG_COV": false,
    }),
  ],
  optimization: {
    minimize: false,
    usedExports: true,
  },
  performance: {
    hints: false,
  },
  stats: {
    colors: true,
  },
  module: {
    rules: [
      {
        test: /\.js$/,
        exclude: /(node_modules)/,
        sideEffects: false,
        use: {
          loader: "babel-loader",
          options: {
            comments: false,
            cacheDirectory: true,
            presets: [
              [
                "@babel/preset-env",
                {
                  targets: { node: "current" },
                  modules: false,
                  debug: false,
                  shippedProposals: true,
                  useBuiltIns: "usage",
                  corejs: "3.9",
                },
              ],
            ],
          },
        },
      },
    ],
  },
};

@deathemperor
Copy link

I've switched to https://github.com/AnomalyInnovations/serverless-bundle and it makes my life easier.

@zirkelc
Copy link

zirkelc commented Sep 23, 2021

I've switched to https://github.com/AnomalyInnovations/serverless-bundle and it makes my life easier.

Are you using Yarn 2 with workspaces?

@j0k3r
Copy link
Member

j0k3r commented Oct 21, 2022

Hey guys, @magelle improved the support for Yarn Workspace in #1258, can you try using the master of serverless-webpack before we cut a new release?

@j0k3r j0k3r added this to the 5.11.0 milestone Nov 22, 2022
@j0k3r j0k3r closed this as completed Nov 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests