[Bug?]: Module not found: Error: Can't resolve 'x' in 'y' packageExtensions binaries [arm, android win32-x64, darwin, gnu, linux]

[gruckion]

Self-service

• I'd be willing to implement a fix

Describe the bug

Mongodb has dependencies that are not listed as peer dependencies. I tried to use packageExtensions to get around this but I am now getting dozens more errors from two libraries Snappy and @mongodb-js/zstd.

➤ YN0000:   in ../../.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd/index.js 89:28-70
➤ YN0000:   Module not found: Error: Can't resolve './zstd.linux-arm-gnueabihf.node' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd'

➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:622-662
➤ YN0000:   Module not found: Error: Can't resolve '@napi-rs/snappy-android-arm64' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'

It looks like it's trying to require all these different binaries for different machines.

My packageExtensions is below and it's managed to get most of the warnings to shut up.

packageExtensions:
  "mongodb@*":
    dependencies:
      bson-ext: "*"
      kerberos: "*"
      "@mongodb-js/zstd": "*"
      aws4: "*"
      mongodb-core: "*"
      bson: "*"
      node-gyp: "*"
      "mongodb-client-encryption": "*"
    peerDependenciesMeta:
      "@mongodb-js/zstd":
        optional: true
      snappy:
        optional: true
      bson-ext:
        optional: true
      kerberos:
        optional: true

To reproduce

I created a repo here, see branch mongodb-broken-pnp.

https://github.com/gruckion/yarn-berry-serverless-workspaces/tree/mongodb-broken-pnp

Run yarn; yarn package to reproduce the errors.

❯ yarn package
➤ YN0000: 
➤ YN0000: Packaging lambdas-hello for stage testing (us-east-1)
➤ YN0000: 
➤ YN0000: Warning: in ../../.yarn/__virtual__/mongodb-virtual-904dd890e4/0/cache/mongodb-npm-4.10.0-99903c2c4a-4847fe69b6.zip/node_modules/mongodb/lib/utils.js 993:38-93
➤ YN0000:   Critical dependency: the request of a dependency is an expression
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd/index.js 17:28-65
➤ YN0000:   Module not found: Error: Can't resolve './zstd.win32-x64-msvc.node' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd/index.js 19:28-70
➤ YN0000:   Module not found: Error: Can't resolve '@mongodb-js/zstd-win32-x64-msvc' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd/index.js 35:28-61
➤ YN0000:   Module not found: Error: Can't resolve './zstd.darwin-x64.node' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd/index.js 37:28-66
➤ YN0000:   Module not found: Error: Can't resolve '@mongodb-js/zstd-darwin-x64' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd/index.js 47:28-63
➤ YN0000:   Module not found: Error: Can't resolve './zstd.darwin-arm64.node' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd/index.js 65:28-64
➤ YN0000:   Module not found: Error: Can't resolve './zstd.linux-x64-gnu.node' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd/index.js 67:28-69
➤ YN0000:   Module not found: Error: Can't resolve '@mongodb-js/zstd-linux-x64-gnu' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd/index.js 77:28-66
➤ YN0000:   Module not found: Error: Can't resolve './zstd.linux-arm64-gnu.node' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd/index.js 79:28-71
➤ YN0000:   Module not found: Error: Can't resolve '@mongodb-js/zstd-linux-arm64-gnu' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd/index.js 89:28-70
➤ YN0000:   Module not found: Error: Can't resolve './zstd.linux-arm-gnueabihf.node' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd/index.js 91:28-75
➤ YN0000:   Module not found: Error: Can't resolve '@mongodb-js/zstd-linux-arm-gnueabihf' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:569-607
➤ YN0000:   Module not found: Error: Can't resolve './snappy.android-arm64.node' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:622-662
➤ YN0000:   Module not found: Error: Can't resolve '@napi-rs/snappy-android-arm64' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:811-852
➤ YN0000:   Module not found: Error: Can't resolve './snappy.android-arm-eabi.node' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:867-910
➤ YN0000:   Module not found: Error: Can't resolve '@napi-rs/snappy-android-arm-eabi' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:1159-1198
➤ YN0000:   Module not found: Error: Can't resolve './snappy.win32-x64-msvc.node' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:1213-1254
➤ YN0000:   Module not found: Error: Can't resolve '@napi-rs/snappy-win32-x64-msvc' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:1403-1443
➤ YN0000:   Module not found: Error: Can't resolve './snappy.win32-ia32-msvc.node' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:1458-1500
➤ YN0000:   Module not found: Error: Can't resolve '@napi-rs/snappy-win32-ia32-msvc' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:1651-1692
➤ YN0000:   Module not found: Error: Can't resolve './snappy.win32-arm64-msvc.node' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:1707-1750
➤ YN0000:   Module not found: Error: Can't resolve '@napi-rs/snappy-win32-arm64-msvc' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:1997-2032
➤ YN0000:   Module not found: Error: Can't resolve './snappy.darwin-x64.node' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:2047-2084
➤ YN0000:   Module not found: Error: Can't resolve '@napi-rs/snappy-darwin-x64' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:2231-2268
➤ YN0000:   Module not found: Error: Can't resolve './snappy.darwin-arm64.node' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:2626-2662
➤ YN0000:   Module not found: Error: Can't resolve './snappy.freebsd-x64.node' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:2677-2715
➤ YN0000:   Module not found: Error: Can't resolve '@napi-rs/snappy-freebsd-x64' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:2900-2939
➤ YN0000:   Module not found: Error: Can't resolve './snappy.linux-x64-musl.node' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:2954-2995
➤ YN0000:   Module not found: Error: Can't resolve '@napi-rs/snappy-linux-x64-musl' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:3131-3169
➤ YN0000:   Module not found: Error: Can't resolve './snappy.linux-x64-gnu.node' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:3184-3224
➤ YN0000:   Module not found: Error: Can't resolve '@napi-rs/snappy-linux-x64-gnu' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:3389-3430
➤ YN0000:   Module not found: Error: Can't resolve './snappy.linux-arm64-musl.node' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:3445-3488
➤ YN0000:   Module not found: Error: Can't resolve '@napi-rs/snappy-linux-arm64-musl' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:3626-3666
➤ YN0000:   Module not found: Error: Can't resolve './snappy.linux-arm64-gnu.node' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:3681-3723
➤ YN0000:   Module not found: Error: Can't resolve '@napi-rs/snappy-linux-arm64-gnu' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:3876-3920
➤ YN0000:   Module not found: Error: Can't resolve './snappy.linux-arm-gnueabihf.node' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:   
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:3935-3981
➤ YN0000:   Module not found: Error: Can't resolve '@napi-rs/snappy-linux-arm-gnueabihf' in '/Users/sigex/workdir/serverless-workspaces-pnp/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000: Environment: darwin, node 16.13.2, framework 3.22.0 (local), plugin 6.2.2, SDK 4.3.2
➤ YN0000: Docs:        docs.serverless.com
➤ YN0000: Support:     forum.serverless.com
➤ YN0000: Bugs:        github.com/serverless/serverless/issues
➤ YN0000: 
➤ YN0000: Error:
➤ YN0000: Webpack compilation failed:
➤ YN0000: 
➤ YN0000: in ../../.yarn/unplugged/@mongodb-js-zstd-darwin-arm64-npm-1.0.0-df74982a20/node_modules/@mongodb-js/zstd-darwin-arm64/zstd.darwin-arm64.node 1:0
➤ YN0000:   Module parse failed: Unexpected character '�' (1:0)
➤ YN0000:   You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
➤ YN0000:   (Source code omitted for this binary file)
➤ YN0000:   
➤ YN0000:   in ../../.yarn/unplugged/@napi-rs-snappy-darwin-arm64-npm-7.1.2-605592bf69/node_modules/@napi-rs/snappy-darwin-arm64/snappy.darwin-arm64.node 1:0
➤ YN0000:   Module parse failed: Unexpected character '�' (1:0)
➤ YN0000:   You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
➤ YN0000:   (Source code omitted for this binary file)
➤ YN0000: 
➤ YN0000: 1 deprecation found: run 'serverless doctor' for more details
➤ YN0000: Done in 17s 69ms

Environment

System:
    OS: macOS 13.0
    CPU: (10) arm64 Apple M1 Max
  Binaries:
    Node: 16.13.2 - /private/var/folders/mv/dtw39jks6ml64s3pd1pnv_640000gn/T/xfs-7dfef409/node
    Yarn: 3.2.3 - /private/var/folders/mv/dtw39jks6ml64s3pd1pnv_640000gn/T/xfs-7dfef409/yarn
    npm: 8.1.2 - ~/.nvm/versions/node/v16.13.2/bin/npm

Additional context

Mongodb has created 3 PRs to "support PnP", but have written no documentation. But this issue is not specific to them, it is more about how we handle libraries that have missing peer dependencies that happen to be binaries.

https://github.com/mongodb/node-mongodb-native/pulls?q=pnp

I also posted a comment on this PR as Mongodb and Mongodb'sissue section on Jira and on Snappy's library

mongodb/node-mongodb-native#2606
https://jira.mongodb.org/browse/NODE-4659
Brooooooklyn/snappy#93

I am also using pnp-webpack-plugin

[yarnbot]

Hi! 👋

This issue looks stale, and doesn't feature the reproducible label - which implies that you didn't provide a working reproduction using Sherlock. As a result, it'll be closed in a few days unless a maintainer explicitly vouches for it or you edit your first post to include a formal reproduction (you can use the playground for that).

Note that we require Sherlock reproductions for long-lived issues (rather than standalone git repositories or similar) because we're a small team. Sherlock gives us the ability to check which bugs are still affecting the master branch at any given point, and decreases the amount of code we need to run on our own machines (thus leading to faster bug resolutions). It helps us help you! 😃

If you absolutely cannot reproduce a bug on Sherlock (for example because it's a Windows-only issue), a maintainer will have to manually add the upholded label. Thanks for helping us triaging our repository! 🌟

[merceyz]

Yarn skips linking incompatible dependencies so your bundler is unable to locate them.

To fix this issue you can either change which packages are linked with supportedArchitectures, change your bundler to avoid trying to bundle them, or avoid using native dependencies.