Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

module: increased surface for hazards with require(esm) experimental flag #52173

Closed
WebReflection opened this issue Mar 21, 2024 · 33 comments · Fixed by #54648
Closed

module: increased surface for hazards with require(esm) experimental flag #52173

WebReflection opened this issue Mar 21, 2024 · 33 comments · Fixed by #54648
Labels
feature request Issues that request new features to be added to Node.js. loaders Issues and PRs related to ES module loaders

Comments

@WebReflection
Copy link
Contributor

WebReflection commented Mar 21, 2024

What is the problem this feature will solve?

In this MR #51977 it's being proposed to allow CJS to require(esm) with the goal of helping people stuck in CJS to use ESM only modules, somehow conflicting with the plethora of dual modules already published and maintained by authors (and I am one of them).

To understand the issue there's no better way than a concrete use case.


Where we are now ...

A project using module a as dependency, where a is published as dual module, can be consumed from pure CJS, where by pure I mean no dynamic import(a) in the mix, as that usually undesired or not common in CJS land due lack of TLA, as well as pure ESM.

module a

// a.mjs
export default Math.random();

// a.cjs
module.exports = Math.random();

module b

console.log(require("a")); // ./a.cjs
// 0.123456789

In this CJS scenario the module a will always provide the same random number once, no matter how many modules require it.

Now, because it was not possible before to synchronously import module c, the module b has no hazards in a CJS only environment.

Now enters module c as ./c.mjs

// module c depends on module `a`
// and because it's ESM only it will
// consume module `a` as ESM
import random from "a"; // ./a.mjs

console.log(random);
// 0.234567891

export default random * 2;

Where we're potentially going ...

edit test the use case if you want

In a scenario where the flag lands "unflagged" and the CJS is still the default, developers will carelessly believe they can finally require(esm) without thinking twice about possible consequences ... if no error is encountered due TLA in the required ESM module, they think they're good!

module b after the flag

console.log(require("a")); // ./a.cjs
// 0.123456789

console.log(require("c").default); // ./c.mjs -> ./a.mjs
// NOT 0.246913578
// BUT 0.469135782

Conclusion

It is true that hazards related to dual modules where already possible before but there was no easy way to synchronously require ESM modules so that either they chose dual modules, they used a tool able to normalize everything as CJS or ESM, but any re-published package as CJS would've avoided or inlined somehow ESM modules with TLA.

In short, if the default require, when it comes to dual modules, still prefers CJS once this flag lands unflagged, we will see dual module authors blamed for issues that were not so easy to bring in before with potentially catastrophic results beyond the dummy Math.random() use case: databases, cache, file system, workers, you name it ... bootstrapping with ease dual modules will likely create more damage than solve instead anything it's aiming to solve and currently published dual modules cannot just stop providing their CJS counterpart until all users and dependents modules are capable of requiring ESM out of the box.

What is the feature you are proposing to solve the problem?

Before proposing anything I don't understand how this experimental flag is being considered as shippable in its current state (broken ESM require if TLA is used behind, increased dual module hazards surface) instead of fixing at the root level the issue by:

  • allow ESM as default in NodeJS
  • allow TLA in CJS too so that dynamic imports won't scare anyone anymore and the TLA behind the scene would also just work

Back to this flag though, I would like to propose at least the following solutions:

  • the default require(anything), once this flag lands unflagged, is to return the ESM version of the module, if such module is a dual module. This would solve the presented use case / issue because the module b that require("a") will already have the ESM version so that once module b finally can also require("c") from the ESM only world, nothing will break and no hazard will be present
  • there's gonna be a flag to impose the preferred way to require a module from CJS so that dual module authors can explicitly indicate in their package.json that the preferred way to require that module is through its ESM code and not its CJS artifact (or vice-versa whenever that's the case). In this case I believe having the "type": "module" in the package.json of a SHOULD already be enough to tell the new require(esm) ability that even when require(cjs) was used, and that module is dual module, the returned module is actually the ESM one.

The latter point was breaking before so it should be a no-brainer to consider while the former suggestion might break with default exports that were not expected before but I am hear to discuss possible solutions there too or propose we give developers time to test and adjust their code, after all they need to do so anyway the moment they finally require(ESM).

Having ESM as preferred way to any require will eventually convince developers to publish ESM only modules so that the dual module story can fade away in time, but if that's not the case I see a catch 22 like situation where authors can't stop publishing dual modules and users can't stop worrying about possible hazards introduced by the new feature.

What alternatives have you considered?

As already mentioned, I would rather bring in TLA in CJS behind a flag instead and call it a day, without mixing a tad too much the require ability of the legacy, non standard, module system that CJS is. This would be the easiest migration for everyone caring to migrate and it will still grant pure CJS software to work without worrying about hazards in the graph.

Thank you.


After Thoughts

It's OK to explore solutions to the problem (we're doing that already) but if NodeJS lands these kind of changes it better be sure that the entirety of the tooling based projects out there also get the memo and implement the right thing or we'll create a new kind of hell to deal with, explain, and resolve ... at least until everything is ESM only and CJS can be just a memory to find in outdated books.

@WebReflection WebReflection added the feature request Issues that request new features to be added to Node.js. label Mar 21, 2024
@nicolo-ribaudo
Copy link
Contributor

(thanks for moving the discussion from Twitter to GitHub!)

the default require(anything), once this flag lands unflagged, is to return the ESM version of the module, if such module is a dual module.

When Joyee tried to make require() accept the import condition rather than just require(), we found that it breaks one of the top-30 ESM packages that she tested (#51977 (comment)).

I'm a maintainer of that package so I could easily fix that, but the problem boils down to the require and import versions not having exactly the same API: the CJS version has a __esModule: true export while the ESM version doesn't. #52166 would fix it, so maybe it's worth considering again the breaking change.

Another case where you would not want require() to respect the import condition is when you have a sync CJS entrypoint and an async ESM entrypoint (for example, because they both do test ? require("./variant-a") : require("./variant-b") and test ? await import("./variant-a") : await import("./variant-b")). In this case, if require() resolved to the ESM version it would then start throwing while loading it. Again, there is a possible solution for package authors: given that conditions are matched in the order they are defined in package.json, you could choose if you want to give priority to import or require:

Priority to CJS Priority to ESM
{
  "name": "foo",
  "exports": {
    "require": "./foo.cjs",
    "import": "./foo.mjs"
  }
}
{
  "name": "foo",
  "exports": {,
    "import": "./foo.mjs",
    "require": "./foo.cjs"
  }
}

This ability to choose priority applies both in the case of just extending require to also look at the import condition, and in the case of just adding a new condition such as require-esm.


Now, because it was not possible before to synchronously import module c, the module b has no hazards in a CJS only environment.

The require(esm) change does not affect CJS-only environments, since it's about mixing CJS and ESM. In mixed environments, such as the example you are giving, the hazard is already present using only synchronous code: import ... from declarations and require() are enough to trigger it (see the example in #52174).

Having ESM as preferred way to any require will eventually convince developers to publish ESM only modules so that the dual module story can fade away in time

Regardless of require() resolution priority, dual modules can be dropped exactly in the moment all the Node.js versions you care about support require(esm), right?

@WebReflection
Copy link
Contributor Author

WebReflection commented Mar 21, 2024

given that conditions are matched in the order they are defined in package.json, you could choose if you want to give priority to import or require

is this a theoretical or that's actually implemented in that flag already? I see your links are still open here and there

In mixed environments, such as the example you are giving, the hazard is already present using only synchronous code

there's no hazard if the first require("a") points at ./a.mjs instead ... dual module per se is not an hazard, is the hybrid env that imports here and there stuff differently that causes the hazard ... here I am suggesting to pivot to ESM always on require whenever that's possible. The change / order of the explicit import works though, I still need to update tons of modules but that's already something, if it works the way you suggested.

dual modules can be dropped exactly in the moment all the Node.js versions you care about support require(esm), right?

it'll take years, I suppose, but yes. New modules won't be published as dual, old modules with hundred million weekly downloads aren't so "easy-peasy" to change.

@nicolo-ribaudo
Copy link
Contributor

is this a theoretical or that's actually implemented in that flag already? I see your links are still open here and there

It's reality that export conditions are matched in order, it's theoretical that require() could match the import condition.

there's no hazard if the first require("a") points at ./a.mjs instead

I agree, but that's not what already happens today without the flag, so today we are already affected by the problem even without using dynamic import (again, see the docs issue I linked because it contains an example :) )

@WebReflection
Copy link
Contributor Author

WebReflection commented Mar 21, 2024

@nicolo-ribaudo this might be a hell of a lucky coincidence but thanks to the fact I use npx modulestrap to bootstrap any project of mine it looks like all my dual packages are already like this:

  "exports": {
    ".": {
      "types": "./types/index.d.ts",
      "import": "./esm/index.js",
      "default": "./cjs/index.js"
    },
    "./package.json": "./package.json"
  },

Am I correct in understanding you that this means there's literally nothing I should do to change because require("flatted"), as example, or require("linkedom") or others will automatically prefer to import the ESM version, as that's first (well, after types, eventually, but that's not a NodeJS affair) in the package export definition?

I might rest my case if that's the case, still if this is how that flag lands everyone should be aware of this potential issue and be sure modules authors also understand how to solve it ... they have time to just eventually swap export definitions without breaking anything in the meantime, the breaking will eventually land once that flag becomes the default.

@nicolo-ribaudo
Copy link
Contributor

If require matched also the import condition then yes, you would would have nothing to change because import would be the first matched condition in your list.

@WebReflection
Copy link
Contributor Author

WebReflection commented Mar 21, 2024

so, the current state is that given this exports definition in the module package that is being either imported or required:

{
  "name": "whatever",
  "type": "module",
  "exports": {
    ".": {
      "types": "./types/index.d.ts",
      "import": "./esm/index.js",
      "default": "./cjs/index.js"
    },
    "./package.json": "./package.json"
  }
}

once this flag gets unflagged the only file that will ever be required or imported is ./esm/index.js ... right?

edit explicitly:

// ./cjs/index.js -> exports.ref = {};
// with flag this points at ./esm/index.js instead
const { ref: cjs } = require("whatever");

// ./esm/index.js -> export const ref = {};
import { ref as esm } from "whatever";

// nothing to see here, it's all good!
console.assert(cjs === esm, "hazards in the house");

If this is the case then my proposal about signaling from module authors what should be preferred is already satisfied ... I just want to be sure this is the case so that at least I don't need to worry about this breaking change in the near future, thank you.

@nicolo-ribaudo
Copy link
Contributor

nicolo-ribaudo commented Mar 21, 2024

No, the behavior of that package doesn't currently change under the flag.

What I was proposing above is to instead explore make the flag break backwards compatibility, so that in your example both require and import would match the import condition and that console.assert (that currently, even without this flag, fails) would pass.

@WebReflection
Copy link
Contributor Author

WebReflection commented Mar 21, 2024

that currently, even without this flag, fails

of course it does ... that was the whole point, I want it to not fail and force require("whatever") to point at ./esm/index.js instead ... can we agree module authors should have this ability? If not, this flag is going to be hostile for users, they add hazards without understanding what's going on, and for module authors, they get the blame for something they can't even control within the Open Source software they provide.

I hope we agree here that make authors able to chose what "future require" should prefer must be a requirement for this flag.

@WebReflection
Copy link
Contributor Author

WebReflection commented Mar 21, 2024

btw ...

that currently, even without this flag, fails

if your point is that it should have a createRequire upfront it's fine ... I wanted to meta describe the desired result from all worlds, no matter if that whatever module is required or imported ... I hope that clarifies.

@nicolo-ribaudo
Copy link
Contributor

nicolo-ribaudo commented Mar 21, 2024

if your point is that it should have a createRequire upfront it's fine

No, my point is that today, in sync-only-code, there is no guarantee that a dual package that uses import or require conditions will be loaded only in one of the two versions. A library depending on yours might be written in CJS and load the CJS version; a library depending on yours might be written in ESM and load the ESM version; and the final users importing those two libraries will see both versions of yours being loaded.


can we agree module authors should have this ability?

Yes, I agree! I disagree that this is a breaking change introduced by the flag, but I agree that module authours should have this ability. And I see four concrete ways to do so, I'll list them here.

Do nothing

This might seem counter intuitive, but I just realized that even if Node.js does nothing you can still publish packages that have the behavior you want.

You need to prefix your transpiled CJS files with a require() of the ESM file:

package.json foo.mjs foo.cjs
{
  "name": "foo",
  "exports": {
    "import": "./foo.mjs",
    "default": "./foo.cjs"
  }
}
let num = Math.random();
export { num };
try {
  return require("./foo.mjs")
} catch {}

let num = Math.random();
exports.num = num;     

Add a new module-sync condition that is matched both by import and require, in platforms where require() can load sync ESM

This would let you have different entrypoints for ESM with TLA (only loaded by import), ESM without TLA (loaded both by import and require, and CJS.

{
  "name": "foo",
  "exports": {
    "module-sync": "./foo.mjs",
    "default": "./foo.cjs"
  }
}
let num = Math.random();
export { num };
let num = Math.random();
exports.num = num;

Add a new require-esm condition that is matched by require in platforms where require() can load sync ESM

{
  "name": "foo",
  "exports": {
	"import": "./foo.mjs",
    "require-esm": "./foo.mjs",
    "default": "./foo.cjs"
  }
}
let num = Math.random();
export { num };
let num = Math.random();
exports.num = num;

Change require() to also match the "import" condition

This is a breaking change, but maybe it could be explored.

@WebReflection
Copy link
Contributor Author

WebReflection commented Mar 21, 2024

P.S. I wouldn't mind a situation like this neither, as ordered JSON is also something not many care about in the wild (AFAIK)

{
  "name": "whatever",
  "type": "module",
  "exports": {
    ".": {
      // edit: oops, taken, I meant require-esm
      "require": "./esm/index.js",
      "types": "./types/index.d.ts",
      "import": "./esm/index.js",
      "default": "./cjs/index.js"
    },
    "./package.json": "./package.json"
  }
}

This adds a new field, it's "scoped" per each defined export, and extremely easy to reason about to me ... we can debate if "cjs" instead of "require-esm" is a better choice, or literally any other convention to satisfy the goal, but I hope something happens that allows authors to decide themselves what's the best qay to require their own modules.

In my case, when no hazards or side effects are possible (i.e. just utilities and who cares if duplicated), I could decide myself which path should be served and prefer maybe CJS if there's only a default export, translated directly as module.exports = thing without all the ugly __esmModule and oter decorations too many tools add these days to solve the same issue.

This would probably be my best option yet because it's also easier for the eyes to understand, and easier for everyone to reason about, imho.

@nicolo-ribaudo
Copy link
Contributor

nicolo-ribaudo commented Mar 21, 2024

That package.json is incompatible with every Node.js version between 12 and <last-version-without-the-flag-enabled-by-default>.

EDIT Oh yes, with a new name it's like the require-esm I proposed above.

@WebReflection
Copy link
Contributor Author

WebReflection commented Mar 21, 2024

No, my point is that today, in sync-only-code, there is no guarantee that a dual package that uses import or require conditions will be loaded only in one of the two versions.

nowhere in this discussion I am mentioning today ... we all know the today story but it looks we're aligned with possible resolutions. I wouldn't use cryptic or counter-intuitive names for DX sake, require-esm sounds good, module-sync sounds weird and non descriptive of its purpose. If you have strong opinion around that name, works for me, I'll write whatever, but please consider keeping it simple in naming too.

About this:

That package.json is incompatible with every Node.js version between 12

I don't care supporting dead LTS and apparently neither do my modules' users (example).

All my dual modules are published like that, we're good to me.

I misunderstood the point in case ... then require is a no-go, agreed, let's find something similar and try to move forward or land it, thank you!

@nicolo-ribaudo
Copy link
Contributor

require is already a valid condition, so it's not just about dead LTSs. It's about Node.js 20, Node.js 22.

@WebReflection
Copy link
Contributor Author

require is already a valid condition, so it's not just about dead LTSs. It's about Node.js 20, Node.js 22.

apologies then that one I didn't know ... so require-esm or require-next or others work to me ... I didn't mean to break even more stuff.

@WebReflection
Copy link
Contributor Author

To whom it might concern, I've created a test-case repository that should not fail in either ESM or CJS with the flag enabled:
https://github.com/WebReflection/examples/tree/main/require-esm

Once that works by just changing "type": "module" to "type": "commonjs" in the main package.json, with or without the flag, we can consider this feature request satisfied.

@joyeecheung
Copy link
Member

joyeecheung commented Mar 21, 2024

Am I understanding correctly that the conclusion here is that we can just add a require-esm condition to allow package authors to indicate a preferred ESM export, for newer versions of Node.js that supports require(esm)? That sounds similar to what my instincts suggested in https://twitter.com/JoyeeCheung/status/1770529091927642232 and should be fairly easy to implement.

Regarding the other point about supporting TLA in CJS in the OP, would you mind split that out to a different issue? I have the impression that there are already existing feature requests and it would be a lot more technically challenging to implement, considering that:

  1. CJS module loader does not separate resolution from evaluation. Which means evaluation of TLA would block the resolution of the next module being require()'d, and requires re-entering the event loop inside the module loader, which is unsafe. Either this has to mismatch with what happens with TLA in ESM, or this has to break the coupling of resolution and evaluation of CJS, which many user land tooling depends on.
  2. All the safety concerns about require(esm) that supports TLA in [WIP] Support requiring .mjs files #30891 applies again to require(cjs) that supports TLA, if not worse, because the CJS loader is monkey patched heavily in the ecosystem. We can't even remove underscored properties from the CJS loader (e.g. Module.prototype._compile) or change the order in which the underscored functions are called, or change their parameters & return types, for example.
  3. De-async in require() likely requires a significant amount of rearchitecting of the CJS loader. Which again would be difficult due to widespread monkey patching.
  4. TLA is not that common in user land modules, so it's not a very cost-effective thing to add additional support in CJS. In my research into the top npm packages, none of the ~30 that I've looked into used TLA. And I was already trying to diversify my sample and skipped a lot of modules that looked similar to some that were already in my sample - none of them contained TLA, either. It's also worth noting that even in ESM, TLA + cycles can lead to deadlocks, or the code below TLA may never run when there is a cycle, so TLA in non-entry-points is already more or less a hazard in itself.

So I think it's best to decouple an issue that's technically simple to address (require-module) from a more or less unrelated issue that may require full rearchitecting + breaking the ecosystem + unsafe re-entrace of the event loop (supporting TLA in CJS).

@WebReflection
Copy link
Contributor Author

WebReflection commented Mar 21, 2024

@joyeecheung

Am I understanding correctly that the conclusion here is that we can just add a require-esm condition to allow package authors to indicate a preferred ESM export, for newer versions of Node.js that supports require(esm)?

That's correct, and if you want to name it require-module works for me, as it narrows down the terminology to even look for out of search engines ... make it require-module to me, "you have my axe"!

Regarding the other point about supporting TLA in CJS in the OP ...

I don't care a single bit about that discussion happening in here, the template to fill out proposals ask alternatives so I've mentioned that alternative but as you pointed out it's been discussed ad nauseam and in my opinion that's the solution to everything in the "migrating CJS to ESM" land but then again, here I am discussing a flag that more than a developer either welcomed or decided it should go in unflagged sooner than later ... hence my loud concerns about the how and the expectations from a module author point of view.

We can already ditch that conversation and I am sorry you felt the need, and spent time, to write down that great summary around that issue ... I personally don't care about that issue if impractical, I do care about this new idea/issue if also impractical for modules' authors.

I hope this reasons well with you.

@bmeck
Copy link
Member

bmeck commented Mar 21, 2024

This problem seems to already exist if C were either type of module and did import('a').

I'm a bit skeptical of require-esm being a good condition; I think TLA condition might be better and more universal as a means to detect if you are in this scenario. Certainly there are ESM only scenarios where knowing if TLA is in use/may be used is good to know even outside this scenario.

@WebReflection
Copy link
Contributor Author

This problem seems to already exist if C were either type of module and did import('a').

If you read carefully that's the thing that doesn't happen at all in the real-world and the thing commented as "of course it was possible already" so please do take time to read the whole thing and understand the real-world concern, thank you!

@WebReflection
Copy link
Contributor Author

WebReflection commented Mar 21, 2024

I think TLA condition might be better and more universal as a means to detect if you are in this scenario

I have no idea what TLA condition is but all I think is: "why not both?" TLA condition can be handy for everything toolchain related and introspection and the require-esm field, or require-module one, would give constraints nobody can escape from unless in outdated NodeJS versions, where the increased hazard surface is nowhere an issue (and at least with my modules, nowhere to find as real-world issue).

@WebReflection
Copy link
Contributor Author

Please Consider ...

I understand somebody took time to analyze top 30 npm modules but maybe we should not forget npm is the biggest repository that ever was and still is on both the backend side and the front end one.

Please do not constrain your assumptions out of top 30 modules and think a bit more broadly, if possible, thank you.

@joyeecheung
Copy link
Member

Please do not constrain your assumptions out of top 30 modules and think a bit more broadly, if possible, thank you.

To quote myself:

And I was already trying to diversify my sample and skipped a lot of modules that looked similar to some that were already in my sample - none of them contained TLA, either.

If we are talking about the packages I've examined, a closer estimate would probably be around top 200-300 - I skimmed the code of a lot of packages, and if they look vastly similar to others I've seen, I didn't include them in my script.

@bmeck
Copy link
Member

bmeck commented Mar 21, 2024

If you read carefully that's the thing that doesn't happen at all in the real-world and the thing commented as "of course it was possible already" so please do take time to read the whole thing and understand the real-world concern, thank you!

I did try to read it and just now re-skimmed trying to find the relevant part, but perhaps I'm confused still

@WebReflection
Copy link
Contributor Author

@joyeecheung it wasn't directed to you, it was a general "please consider" statement ... but on top 30 modules I don't know if FE modules are included but on the Web TLA is expected, it works well, and it's desired.

The fact people even think TLA was a mistake is a concern I don't want to even start discussing in here as that involves way more standard groups than just NodeJS.

@WebReflection
Copy link
Contributor Author

WebReflection commented Mar 21, 2024

I did try to read it and just now re-skimmed trying to find the relevant part, but perhaps I'm confused still

So if you follow, nobody is using dynamic import("c") to date, in my example, because that's not practical for a pure CJS environment, so they never needed, or wanted, to use a "c" module that was only ESM.

This flag allows them to not think about it and finally require("c") so that all the guards behind the hazard are unleashed like there's no tomorrow.

It was possible before but if analysis is what we are after, and the reason that flag exists at all, is that CJS stuck folks really want to synchronously import ESM and that opens a can of warms all over the graph because those ESM only modules might depend on dual modules behind the scene, and in doing so, they already bring in ESM instead of CJS versions that the previous pure/CJS only codebase already used, and enjoyed, as dual module.

@WebReflection
Copy link
Contributor Author

WebReflection commented Mar 21, 2024

to @bmeck or others, if you reverse engineer this, it actually makes sense for any ESM only module that depend on a dual module to instead be somehow "poisoned" and import the CJS version of that dual module to satisfy the current main project asking require to import that ESM only module ... yeah, this is a way forward to me: if require-module or require-sem brings in an ESM only module, that module in that graph should be flagged as one preferring the old style require as CJS when any of its imports point at a dual module. In that case the hazard is mitigated too ... so that's also an upside-down twisted alternative to me, as long as the hazard is not backed in core NodeJS.

edit I understand this is complicate ... but to me it's an option that might work for the CJS crew and produce the least amount of shenanigans ... if the source of the import/require is CJS, bring in all CJS you can, bring in all ESM you can otherwise

@joyeecheung
Copy link
Member

So if you follow, nobody is using dynamic import("c") to date, in my example, because that's not practical for a pure CJS environment, so they never needed, or wanted, to use a "c" module that was only ESM.

FYI, even in our dependencies...

node/deps/npm/lib/npm.js

Lines 200 to 201 in af8ba37

import('chalk'),
import('supports-color'),

And from the amount of support issues regarding importModuleDynamically I suspect enough people have been using it already e.g. #35889 (comment)

@joyeecheung
Copy link
Member

joyeecheung commented Mar 21, 2024

if you reverse engineer this, it actually makes sense for any ESM only module that depend on a dual module to instead be somehow "poisoned" and import the CJS version of that dual module to satisfy the current main project asking require to import that ESM only module ... yeah, this is a way forward to me: if require-module or require-sem brings in an ESM only module, that module in that graph should be flagged as one preferring the old style require as CJS when any of its imports point at a dual module.

I think you are talking about #52174 ? For example, if a dual module makes itself dual by defining both require and import in its conditional exports, this can happen:

C (ESM) -> B(CJS) -> A(dual: CJS)
C (ESM) -> A (dual: ESM)

This is unrelated to require(esm), since you could already import cjs or createRequire(import.meta.url)(cjs) (and not that uncommon, actually).

But if you follow the suggestion in #52174 and make CJS the default of dual packages in Node.js, this happens:

C (ESM) -> B(CJS) -> A(dual: CJS)
C (ESM) -> A (dual: CJS)

And the hazard is gone. So I guess the problem comes down to dual modules going ESM-first in Node.js even though ESM can only be consumed by ESM, thus inviting a branched graph, so this predates require(esm). To prevent the hazard, you'll need to let the format that can be consumed by both ESM and CJS to take precedence (which is currently, CJS). And if you follow the suggestion in #52174, you have a configuration that is safe from the branching of both import cjs and require(esm).

Before require(esm), the only way for you to avoid the hazard is to go CJS-first. After require(esm), without a require-module conditional exports, you should still go CJS-first to avoid the hazard. With a require-module though it opens new possibility for you to go ESM-first safely. Otherwise, ESM-first is just unsafe, with or without require(esm).

@WebReflection
Copy link
Contributor Author

And the hazard is gone. So I guess the problem comes down to dual modules going ESM-first in Node.js

I think once require-module or require-esm goes in I can dictate from my module what that thing should consume if dual which could be either the ESM or the CJS when the hazard is obvious and risky, private module cache is used, or other things such as DB connection via .env variables happen once.

So this require-module or require-esm conditional export seems to solve it all to me and I hope it will be implemented in a way that my example works and the only change I need to do is to add that field in the exports among others and that field gets top priority if available so that hazards would be on me to avoid, it's me deciding if for those cases CJS is better or if everything should simply use ESM when no top level await is used, so that also the dynamic import story doesn't add extra hazards.

Thanks.

@joyeecheung
Copy link
Member

joyeecheung commented Mar 22, 2024

Yes, but at the meantime, before require(esm) or this new export conditions go out of experiment, dual packages still have to do CJS-first for safety.

So until require(esm) and a new conditional export for ESM-first preference in Node.js are shipped, the only way to do dual packages safely is:

"exports": {
  "node": "./cjs.cjs",  // When Node.js doesn't support require(esm), the only way to avoid duplicates is to always use CJS
  "default": "./esm.mjs"
}

After require(esm) and the new conditional export for ESM-first preference are shipped, it unlocks the new possibility to go ESM-first on newer versions of Node.js that support both, and fallback to CJS-first on older versions of Node.js, with something like this:

"exports": {
  "node-esm": "./esm.mjs",  // This gets ignored by older versions of Node.js but can be picked up by newer versions
  "node": "./cjs.cjs",  // This gets picked up by older versions of Node.js
  "default": "./esm.mjs"
}

The precedence depends on the package author because the conditional exports are applied in insertion order.

@joyeecheung
Copy link
Member

joyeecheung commented Mar 22, 2024

In your example, you are doing something like this:

  "exports": {
    ".": {
      "require-esm": "./index.js",
      "import": "./index.js",
      "default": "./index.cjs"
    },

This is already and will always be unsafe on older versions of Node.js, unrelated torequire(esm), because it encourages ESM to load the ESM version of the package, and CJS to load the CJS version of the package. Once there is a ESM that imports CJS that loads the CJS version of the package, you can end up with two versions of the same package in the process.

C (ESM) -> B(CJS) -> A(dual: CJS)
C (ESM) -> A (dual: ESM)

This is unrelated to dynamic import, by the way. The problematic edge is C (ESM) -> B(CJS), which is enabled byimport cjs or createRequire(import.meta.url)(cjs) and is quite common because many ESM-only packages also have CJS-only dependencies.

Let's suppose on v22 we ship require(esm) and a new conditional export, then your package may start becoming safe on v22 since this will start to become possible:

C (ESM) -> B(CJS) -> A(dual: ESM)
C (ESM) -> A (dual: ESM)

But if Node.js never ships require(esm) (probably not likely), then the conditional exports in your example will be unsafe forever. On Node.js versions that don't support require(esm), your only safe option is CJS-first, like this:

C (ESM) -> B(CJS) -> A(dual: CJS)
C (ESM) -> A (dual: CJS)

@avivkeller avivkeller added the loaders Issues and PRs related to ES module loaders label Apr 21, 2024
@GeoffreyBooth
Copy link
Member

@nodejs/loaders

joyeecheung added a commit to joyeecheung/node that referenced this issue Oct 1, 2024
This patch implements a "module-sync" exports condition
for packages to supply a sycnrhonous ES module to the
Node.js module loader, no matter it's being required
or imported. This is similar to the "module" condition
that bundlers have been using to support `require(esm)`
in Node.js, and allows dual-package authors to opt into
ESM-first only newer versions of Node.js that supports
require(esm) while avoiding the dual-package hazard.

```json
{
  "type": "module",
  "exports": {
    "node": {
      // On new version of Node.js, both require() and import get
      // the ESM version
      "module-sync": "./index.js",
      // On older version of Node.js, where "module" and
      // require(esm) are not supported, use the transpiled CJS version
      // to avoid dual-package hazard. Library authors can decide
      // to drop support for older versions of Node.js when they think
      // it's time.
      "default": "./dist/index.cjs"
    },
    // On any other environment, use the ESM version.
    "default": "./index.js"
  }
}
```

We end up implementing a condition with a different name
instead of reusing "module", because existing code in the
ecosystem using the "module" condition sometimes also expect
the module resolution for these ESM files to work in CJS
style, which is supported by bundlers, but the native
Node.js loader has intentionally made ESM resolution
different from CJS resolution (e.g. forbidding `import
'./noext'` or `import './directory'`), so it would be
semver-major to implement a `"module"` condition
without implementing the forbidden ESM resolution rules.
For now, this just implments a new condition as semver-minor
so it can be backported to older LTS.

Refs: https://webpack.js.org/guides/package-exports/#target-environment-independent-packages
PR-URL: nodejs#54648
Fixes: nodejs#52173
Refs: https://github.com/joyeecheung/test-module-condition
Refs: nodejs#52697
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: Jan Krems <jan.krems@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
targos pushed a commit that referenced this issue Oct 4, 2024
This patch implements a "module-sync" exports condition
for packages to supply a sycnrhonous ES module to the
Node.js module loader, no matter it's being required
or imported. This is similar to the "module" condition
that bundlers have been using to support `require(esm)`
in Node.js, and allows dual-package authors to opt into
ESM-first only newer versions of Node.js that supports
require(esm) while avoiding the dual-package hazard.

```json
{
  "type": "module",
  "exports": {
    "node": {
      // On new version of Node.js, both require() and import get
      // the ESM version
      "module-sync": "./index.js",
      // On older version of Node.js, where "module" and
      // require(esm) are not supported, use the transpiled CJS version
      // to avoid dual-package hazard. Library authors can decide
      // to drop support for older versions of Node.js when they think
      // it's time.
      "default": "./dist/index.cjs"
    },
    // On any other environment, use the ESM version.
    "default": "./index.js"
  }
}
```

We end up implementing a condition with a different name
instead of reusing "module", because existing code in the
ecosystem using the "module" condition sometimes also expect
the module resolution for these ESM files to work in CJS
style, which is supported by bundlers, but the native
Node.js loader has intentionally made ESM resolution
different from CJS resolution (e.g. forbidding `import
'./noext'` or `import './directory'`), so it would be
semver-major to implement a `"module"` condition
without implementing the forbidden ESM resolution rules.
For now, this just implments a new condition as semver-minor
so it can be backported to older LTS.

Refs: https://webpack.js.org/guides/package-exports/#target-environment-independent-packages
PR-URL: #54648
Fixes: #52173
Refs: https://github.com/joyeecheung/test-module-condition
Refs: #52697
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: Jan Krems <jan.krems@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
louwers pushed a commit to louwers/node that referenced this issue Nov 2, 2024
This patch implements a "module-sync" exports condition
for packages to supply a sycnrhonous ES module to the
Node.js module loader, no matter it's being required
or imported. This is similar to the "module" condition
that bundlers have been using to support `require(esm)`
in Node.js, and allows dual-package authors to opt into
ESM-first only newer versions of Node.js that supports
require(esm) while avoiding the dual-package hazard.

```json
{
  "type": "module",
  "exports": {
    "node": {
      // On new version of Node.js, both require() and import get
      // the ESM version
      "module-sync": "./index.js",
      // On older version of Node.js, where "module" and
      // require(esm) are not supported, use the transpiled CJS version
      // to avoid dual-package hazard. Library authors can decide
      // to drop support for older versions of Node.js when they think
      // it's time.
      "default": "./dist/index.cjs"
    },
    // On any other environment, use the ESM version.
    "default": "./index.js"
  }
}
```

We end up implementing a condition with a different name
instead of reusing "module", because existing code in the
ecosystem using the "module" condition sometimes also expect
the module resolution for these ESM files to work in CJS
style, which is supported by bundlers, but the native
Node.js loader has intentionally made ESM resolution
different from CJS resolution (e.g. forbidding `import
'./noext'` or `import './directory'`), so it would be
semver-major to implement a `"module"` condition
without implementing the forbidden ESM resolution rules.
For now, this just implments a new condition as semver-minor
so it can be backported to older LTS.

Refs: https://webpack.js.org/guides/package-exports/#target-environment-independent-packages
PR-URL: nodejs#54648
Fixes: nodejs#52173
Refs: https://github.com/joyeecheung/test-module-condition
Refs: nodejs#52697
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: Jan Krems <jan.krems@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
tpoisseau pushed a commit to tpoisseau/node that referenced this issue Nov 21, 2024
This patch implements a "module-sync" exports condition
for packages to supply a sycnrhonous ES module to the
Node.js module loader, no matter it's being required
or imported. This is similar to the "module" condition
that bundlers have been using to support `require(esm)`
in Node.js, and allows dual-package authors to opt into
ESM-first only newer versions of Node.js that supports
require(esm) while avoiding the dual-package hazard.

```json
{
  "type": "module",
  "exports": {
    "node": {
      // On new version of Node.js, both require() and import get
      // the ESM version
      "module-sync": "./index.js",
      // On older version of Node.js, where "module" and
      // require(esm) are not supported, use the transpiled CJS version
      // to avoid dual-package hazard. Library authors can decide
      // to drop support for older versions of Node.js when they think
      // it's time.
      "default": "./dist/index.cjs"
    },
    // On any other environment, use the ESM version.
    "default": "./index.js"
  }
}
```

We end up implementing a condition with a different name
instead of reusing "module", because existing code in the
ecosystem using the "module" condition sometimes also expect
the module resolution for these ESM files to work in CJS
style, which is supported by bundlers, but the native
Node.js loader has intentionally made ESM resolution
different from CJS resolution (e.g. forbidding `import
'./noext'` or `import './directory'`), so it would be
semver-major to implement a `"module"` condition
without implementing the forbidden ESM resolution rules.
For now, this just implments a new condition as semver-minor
so it can be backported to older LTS.

Refs: https://webpack.js.org/guides/package-exports/#target-environment-independent-packages
PR-URL: nodejs#54648
Fixes: nodejs#52173
Refs: https://github.com/joyeecheung/test-module-condition
Refs: nodejs#52697
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: Jan Krems <jan.krems@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request Issues that request new features to be added to Node.js. loaders Issues and PRs related to ES module loaders
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants