[Bug?]: yarn constraints exits with fatal error if yarn.config.cjs imports from ESM, unless executed with DISABLE_V8_COMPILE_CACHE=1 yarn constraints

[milesrichardson]

Hey! Thanks for all your work on Yarn 4. :) I love it so far, and writing constraints in JS is a nice change. I've encountered a bug, but there's a workaround...

---

Self-service

• I'd be willing to implement a fix

Describe the bug

Running yarn constraints throws a fatal error ("A dynamic import callback was not specified") when yarn.config.cjs attempts to import() code from an ECMAScript Module. This is a bug, because dynamic import() should be available in all CommonJS modules (in fact, when you try to require() from an ESM, the error message even tells you to switch to import()).

However, the script executes successfully when executed with DISABLE_V8_COMPILE_CACHE=1 yarn constraints. This indicates the bug is likely in the v8-compile-cache package (possibly the one referenced in this open issue from 2020). However, I have not investigated the root cause. For me, the workaround is fine, and I simply added a script in my package.json to set the variable: "scripts": { "check-constraints": "DISABLE_V8_COMPILE_CACHE=1 yarn constraints" }.

Possibly related: I am using the node-modules linker.

To reproduce

1. Create a new Yarn project with node-modules linker

First, create a new Yarn project. I've noticed this bug with the node-modules linker. I haven't attempted to replicate it with PnP. This is my .yarnrc.yml:

nodeLinker: node-modules

2. Install the is-exact-version package

The package I am attempting to import is is-exact-version, which is installed in the dependencies of my root workspace. You can reproduce it by installing the same version I have:

yarn add -E is-exact-version@2.0.0-beta.1

This is what the main export of that package looks like (note that it's using import statements):

# The main export is a .js file
❯ jq -r '.main' node_modules/is-exact-version/package.json
dist/is-exact-version.js

# It's the only exported .js file
❯ ls node_modules/is-exact-version/dist/
is-exact-version.d.ts   is-exact-version.js     is-exact-version.js.map

# It imports from its dependency semver using ESM syntax
❯ head node_modules/is-exact-version/dist/is-exact-version.js
import clean from 'semver/functions/clean.js';
import Range from 'semver/classes/range.js';
let log = function (...args) { };
(async function optionallyLoadDebugLogger() {
    try {

# It's depending on version 7.3.5 of semver
❯ yarn why semver
...
├─ is-exact-version@npm:2.0.0-beta.1
│  └─ semver@npm:7.5.4 (via npm:^7.3.5)
│
├─ is-exact-version@npm:2.0.0-beta.1 [d00d6]
│  └─ semver@npm:7.5.4 (via npm:^7.3.5)
│

3. Sanity check: require() fails, as expected

Put this into yarn.config.cjs and run yarn constraints:

const { defineConfig } = require("@yarnpkg/types");
const { isExactVersion } = require("is-exact-version");

module.exports = defineConfig({
  constraints: async (ctx) => {
    ctx.Yarn.dependencies()[0].error("The script ran successfully, at least");
    return;
  },
});

As expected, we get an error telling us to import instead:

❯ yarn constraints
Internal Error: require() of ES Module /Users/myusername/path-to/my-project/node_modules/is-exact-version/dist/is-exact-version.js from /Users/myusername/path-to/my-project/yarn.config.cjs not supported.
Instead change the require of is-exact-version.js in /Users/myusername/path-to/my-project/yarn.config.cjs to a dynamic import() which is available in all CommonJS modules.
Instead change the require of is-exact-version.js in /Users/myusername/path-to/my-project/yarn.config.cjs to a dynamic import() which is available in all CommonJS modules.
    at require2 (/Users/myusername/.nvm/versions/node/v20.9.0/lib/node_modules/corepack/dist/lib/corepack.cjs:36309:24)
    at Object.<anonymous> (/Users/myusername/path-to/my-project/yarn.config.cjs:2:28)
    at Module2._compile (/Users/myusername/.nvm/versions/node/v20.9.0/lib/node_modules/corepack/dist/lib/corepack.cjs:36326:34)
    at require2 (/Users/myusername/.nvm/versions/node/v20.9.0/lib/node_modules/corepack/dist/lib/corepack.cjs:36309:24)
    at yne (/Users/myusername/.cache/node/corepack/yarn/4.0.1/yarn.js:140:54306)
    at zp (/Users/myusername/.cache/node/corepack/yarn/4.0.1/yarn.js:140:55036)
    at St.loadUserConfig (/Users/myusername/.cache/node/corepack/yarn/4.0.1/yarn.js:210:5667)
    at async p0.execute (/Users/myusername/.cache/node/corepack/yarn/4.0.1/yarn.js:526:1798)
    at async p0.validateAndExecute (/Users/myusername/.cache/node/corepack/yarn/4.0.1/yarn.js:94:787)
    at async as.run (/Users/myusername/.cache/node/corepack/yarn/4.0.1/yarn.js:98:3250)
    at async iPt (/Users/myusername/.cache/node/corepack/yarn/4.0.1/yarn.js:734:11269)
    at async sk (/Users/myusername/.cache/node/corepack/yarn/4.0.1/yarn.js:734:11625)

4. Reproduce the bug: import() fails

Try the simplest possible dynamic import():

const { defineConfig } = require("@yarnpkg/types");
module.exports = defineConfig({
  constraints: async (ctx) => {
    const thisIsAnEcmaScriptModule = await import("is-exact-version");
    ctx.Yarn.dependencies()[0].error("The script ran successfully, at least");
    return;
  },
});

This fails with error:

❯ yarn constraints
Type Error: A dynamic import callback was not specified.
    at new NodeError (node:internal/errors:406:5)
    at importModuleDynamicallyCallback (node:internal/modules/esm/utils:144:9)
    at Object.constraints (/Users/myusername/path-to/my-project/yarn.config.cjs:4:38)
    at CC.process (/Users/myusername/.cache/node/corepack/yarn/4.0.1/yarn.js:524:73339)
    at async p0.execute (/Users/myusername/.cache/node/corepack/yarn/4.0.1/yarn.js:526:1996)
    at async p0.validateAndExecute (/Users/myusername/.cache/node/corepack/yarn/4.0.1/yarn.js:94:787)
    at async as.run (/Users/myusername/.cache/node/corepack/yarn/4.0.1/yarn.js:98:3250)
    at async iPt (/Users/myusername/.cache/node/corepack/yarn/4.0.1/yarn.js:734:11269)
    at async sk (/Users/myusername/.cache/node/corepack/yarn/4.0.1/yarn.js:734:11625)

4. Reproduce the workaround

Setting DISABLE_V8_COMPILE_CACHE=1 resolves the issue and the script runs successfully:

❯ DISABLE_V8_COMPILE_CACHE=1 yarn constraints
└─ myproject@workspace:.
   └─ The script ran successfully, at least

Appendix: Potential fix that does not work

Given the error about the unresolved callback, I tried wrapping the import safely. This makes no difference - it produces the same error:

const { defineConfig } = require("@yarnpkg/types");

function loadIsExactVersion() {
  return new Promise((resolve, reject) => {
    import("is-exact-version")
      .then((module) => {
        resolve(module.isExactVersion);
      })
      .catch((error) => {
        reject(error);
      });
  });
}

module.exports = defineConfig({
  constraints: async (ctx) => {
    const thisIsAnEcmaScriptModule = await loadIsExactVersion();
    ctx.Yarn.dependencies()[0].error("The script ran successfully, at least");
    return;
  },
});

Environment

❯ yarn dlx -q envinfo --preset jest

  System:
    OS: macOS 13.5
    CPU: (8) x64 Intel(R) Core(TM) i5-1038NG7 CPU @ 2.00GHz
  Binaries:
    Node: 20.9.0 - /private/var/folders/np/djbv9lnn5wd62yrs60zxh_p40000gn/T/xfs-fae2114e/node
    Yarn: 4.0.1 - /private/var/folders/np/djbv9lnn5wd62yrs60zxh_p40000gn/T/xfs-fae2114e/yarn
    npm: 10.1.0 - ~/.nvm/versions/node/v20.9.0/bin/npm

Additional context

As mentioned, I am using the node-modules linker.

The only reason I noticed that DISABLE_V8_COMPILE_CACHE=1 fixed the bug is because when I was switching back and forth between attempts to fix the script, I would sometimes encounter a different error, "invalid host defined options." Googling that led me to v8-compile-cache. There are some recent issues with similar errors in other projects, e.g. pyodide/pyodide#4267 and prettier/prettier-vscode#3114 (Then I asked ChatGPT if I could opt out of v8-compile-cache, and it told me I could set DISABLE_V8_COMPILE_CACHE=1, which sounded so convenient I thought it was hallucinating, but it worked!)

[milesrichardson]

UPDATE: I think the bug is in corepack.

I upgraded Node from 20.9.0 to the latest stable version 20.10.0.

The bug is NOT fixed in 4.0.2 stable

I upgraded yarn from 4.0.1 to the latest stable 4.0.2 version with yarn set version stable. In this case, the bug is NOT fixed:

❯ yarn dlx -q envinfo --preset jest

  System:
    OS: macOS 13.5
    CPU: (8) x64 Intel(R) Core(TM) i5-1038NG7 CPU @ 2.00GHz
  Binaries:
    Node: 20.10.0 - /private/var/folders/np/djbv9lnn5wd62yrs60zxh_p40000gn/T/xfs-414ff96f/node
    Yarn: 4.0.2 - /private/var/folders/np/djbv9lnn5wd62yrs60zxh_p40000gn/T/xfs-414ff96f/yarn
    npm: 10.2.3 - ~/.nvm/versions/node/v20.10.0/bin/npm

❯ yarn constraints
Type Error: A dynamic import callback was not specified.
    at importModuleDynamicallyCallback (node:internal/modules/esm/utils:182:9)
    at /Users/myusername/path-to/my-project/yarn.config.cjs:10:5
    at new Promise (<anonymous>)
    at loadIsExactVersion (/Users/myusername/path-to/my-project/yarn.config.cjs:9:10)
    at Object.constraints (/Users/myusername/path-to/my-project/yarn.config.cjs:22:35)
    at wC.process (/Users/myusername/.cache/node/corepack/yarn/4.0.2/yarn.js:524:74675)
    at async p0.execute (/Users/myusername/.cache/node/corepack/yarn/4.0.2/yarn.js:526:1996)
    at async p0.validateAndExecute (/Users/myusername/.cache/node/corepack/yarn/4.0.2/yarn.js:94:787)
    at async as.run (/Users/myusername/.cache/node/corepack/yarn/4.0.2/yarn.js:98:3250)
    at async oPt (/Users/myusername/.cache/node/corepack/yarn/4.0.2/yarn.js:734:11269)

The bug appears to be fixed in the latest from master, but is it?

The bug appears to be fixed after installing the version of yarn from master using yarn set version from sources.

However, I believe this is an illusion, indicating that the real bug is in corepack, which is not used when installing Yarn from source, as noted in the docs:

Unlike the stable and canary channels, the yarn set version from sources command can't leverage Corepack and will need to store the Yarn binary inside the .yarn/releases folder and reference it from your project's .yarnrc.yml file.

❯ yarn dlx -q envinfo --preset jest

  System:
    OS: macOS 13.5
    CPU: (8) x64 Intel(R) Core(TM) i5-1038NG7 CPU @ 2.00GHz
  Binaries:
    Node: 20.10.0 - /private/var/folders/np/djbv9lnn5wd62yrs60zxh_p40000gn/T/xfs-aca9279f/node
    Yarn: 4.0.2-git.20231115.hash-017b94a - /private/var/folders/np/djbv9lnn5wd62yrs60zxh_p40000gn/T/xfs-aca9279f/yarn
    npm: 10.2.3 - ~/.nvm/versions/node/v20.10.0/bin/npm

❯ yarn constraints
└─ myproject@workspace:.
   └─ The script ran successfully, at least