Node 20 ESM loaders cannot hook into the CommonJS loader

[TomasHubelbauer]

Version

20.1.0

Platform

Darwin Tomass-MBP-2.netis.cc 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar 6 21:00:17 PST 2023; root:xnu-8796.101.5~3/RELEASE_X86_64 x86_64

Subsystem

node --loader

What steps will reproduce the bug?

Follow the steps in my repro here: https://github.com/TomasHubelbauer/node-esm-loader-repro

Copied here for posterity.

Node 20:

1. nvm install 20 to install Node 20
2. node --version to ensure Node version (I get 20.1.0)
3. npm install to install dependencies
4. npm run test to run the health.test.ts script

Notice the test fails and Fastify's autoload is seemingly not aware of the
--loader option and attempts to load routes/health.ts without TypeScript to
JavaScript conversion via ts-node/esm.

npm run test

> repro@0.0.0 test
> node --loader=ts-node/esm --experimental-specifier-resolution=node --test health.test.ts

ℹ (node:95105) ExperimentalWarning: Custom ESM Loaders is an experimental feature and might change at any time
ℹ (Use `node --trace-warnings ...` to show where the warning was created)
✖ should be alive (32.750836ms)
  Error: "@fastify/autoload cannot import plugin at '/routes/health.ts'. To fix this error compile TypeScript to JavaScript or use 'ts-node' to run your app."
      at findPlugins (/node_modules/@fastify/autoload/index.js:224:15)
      at async autoload (/node-esm-loader-repro/node_modules/@fastify/autoload/index.js:35:22)

ℹ tests 1
ℹ suites 0
ℹ pass 0
ℹ fail 1
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 2208.335483

✖ failing tests:

✖ should be alive (32.750836ms)
  Error: "@fastify/autoload cannot import plugin at 'routes/health.ts'. To fix this error compile TypeScript to JavaScript or use 'ts-node' to run your app."
      at findPlugins (/node_modules/@fastify/autoload/index.js:224:15)
      at async autoload (/node_modules/@fastify/autoload/index.js:35:22)

Node 19:

1. nvm install 19 to install Node 20
2. node --version to ensure Node version (I get 19.9.0)
3. npm install to install dependencies
4. npm run test to run the health.test.ts script

Notice the test passes and Fastify's autoload is inherit the --loader option
and uses the ts-node/esm loader successfully to auto-load routes/health.ts.

npm run test

> repro@0.0.0 test
> node --loader=ts-node/esm --experimental-specifier-resolution=node --test health.test.ts

ℹ (node:95453) ExperimentalWarning: Custom ESM Loaders is an experimental feature and might change at any time
ℹ (Use `node --trace-warnings ...` to show where the warning was created)
✔ should be alive (472.711395ms)
ℹ tests 1
ℹ suites 0
ℹ pass 1
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 2679.742161

How often does it reproduce? Is there a required condition?

Node 20: every time
Node 10: never

What is the expected behavior? Why is that the expected behavior?

Follow Node 19's behavior of using --loader to transpile TypeScript to JavaScript on the fly using ts-node/esm.

What do you see instead?

An error that is the result of the loader not being applied.

Additional information

No response

[MoLow]

@TomasHubelbauer does it make a difference if you run with or without --test?

[TomasHubelbauer]

No difference, see this branch: https://github.com/TomasHubelbauer/node-esm-loader-repro/tree/without-test
The readme there contains the error messages I get when dropping --test and changing the script to be top-level code instead of an it.

[targos]

This was reported last week by @isaacs on Slack.

The problem is that ts-node/esm also hooks into the CommonJS loader (require.extensions), but since #44710, it has no effect because it doesn't happen on the main user thread.

[targos]

/cc @cspotcode @nodejs/loaders

[isaacs]

This may be related to #47747. It is clear that the "off-thread loaders" feature seems to have introduced some significant instability, and in my opinion should probably be rolled back and put off until the next major, or at least until there is a straightforward way to allow loaders to either (a) modify require.extensions in such a way that the loaded modules see the modifications, or (b) return transpiled source for "type": "commonjs" modules. If the approach is (b), then that is still a significant breaking change that should get the full cycle of play testing in an unstable v21 release cycle before going live. If the approach is (a), then I think that would solve at least the ts-node/esm problems, which is definitely the most acute community impact.

My preference would be to roll back off-thread loaders in v20, and asap in v21, provide the capability for loaders to return transpiled source for commonjs modules. That would give existing transpilers plenty of time to migrate to the new approach, and also unify the esm/cjs story considerably, and provide a path to finally fully deprecating require.extensions.

[JakobJingleheimer]

I don't foresee a rollback to accommodate a small subset of use-cases that happens to affect you. I do think enabling loaders to supply source for commonjs is a near-future priority (which I intend to champion). Enabling loaders to supply source for commonjs is most definitely not a breaking change: It was very explicitly not supported—if a user happened to erroneously try and node didn't blow up in their face for doing what was specifically detailed as not supported, congratulations to whoever for foot-gunning themselves.

This change was 7 months in the making, and was well publicised.

I am very content to facilitate and support transition. If an author needs time, inform me/us. To my knowledge, we received no requests for such, and during the time, several packages were updated.

I believe the issue here is a known bug which has a simple workaround. The bug as I understand it should indeed be addressed, and we plan to do.

As a side note, I am personally very offended by this cavalier and callous suggestion. I made every effort to forewarn and collaborate with package authors (I specifically tagged major packages), and in several cases, made the compatibility updates for them.

We had discussed this "privately", and I suggested the resolution of enabling loaders supplying commonjs source, so I do not appreciate this public sucker-punch.

[GeoffreyBooth]

This may be related to #47747. It is clear that the “off-thread loaders” feature seems to have introduced some significant instability, and in my opinion should probably be rolled back and put off until the next major

No. The Loaders API is experimental. Breaking changes are expected.

The intention is to add support for the load hook to return source for type: "commonjs". That’s listed here: https://github.com/nodejs/loaders#milestone-2-stability. That’s one of the few remaining items before we consider the API to be stable.

The team working on this part of the codebase has very limited resources. If you’d like to help, your contribution would be most welcome.

[GeoffreyBooth]

The problem is that ts-node/esm also hooks into the CommonJS loader (require.extensions), but since #44710, it has no effect because it doesn’t happen on the main user thread.

An immediate solution is to run the require.extensions registration on the main thread in addition to registering the loader. Something like --import ts-node/register --loader ts-node/esm where ts-node/register includes the require.extensions logic that’s currently bundled into the loader.

[isaacs]

@GeoffreyBooth yes, that is a workaround for users. But more likely, they'll just be annoyed that node 20 mysteriously broke their builds.

[isaacs]

This may be related to #47747. It is clear that the “off-thread loaders” feature seems to have introduced some significant instability, and in my opinion should probably be rolled back and put off until the next major

No. The Loaders API is experimental. Breaking changes are expected.

"Breaking changes", in the sense of changes that require consumers to adjust their consuming code, as in a break from the past API to a new thing, yes. "Just broken, without any added benefit," no. That's not what "breaking changes" means, and "experimental" is not a license to ship a broken program.

The purpose of an "experimental" API is to gain information. It has done that, successfully. The information coming back from users is "this is a change that, on its own, without providing any other way to transpile CJS, makes the loaders API worse for users". The right thing to do is to take that information and act accordingly.

The intention is to add support for the load hook to return source for type: "commonjs". That’s listed here: https://github.com/nodejs/loaders#milestone-2-stability. That’s one of the few remaining items before we consider the API to be stable.

Great! I agree this is the correct goal.

The team working on this part of the codebase has very limited resources.

All the more reason to revert it in v20, and continue the work in v21, in the hopes of shipping a more complete implementation in the future, without the burden of having to keep repeating this conversation for the duration of v20's support lifetime.

If you’d like to help, your contribution would be most welcome.

I would be happy to. Who is this team? What's the best way to engage with them?

This is a blocker for a project I currently care a lot about, so I'm motivated to see it addressed. I am willing and able to dive in to solve the problem. But I'm not super eager to waste time on political wrangling or bikeshedding. If getting CJS transpilation into v20 is not going to be feasible in the short term, and reverting the off thread loading is out of the question, then I'll just drop support for v20 until it's taken care of.

[isaacs]

This change was 7 months in the making, and was well publicised.

It did not appear in any v19 release. I don't think it was as well publicized as you think. It has taken the vast majority of TypeScript users by surprise.

[isaacs]

@JakobJingleheimer It was not my intent for this to be a "public sucker punch". I'm coming at this just looking at the state of the code, and what it means for consumers. This is extremely not personal.

The fact is, this broke the ts-node esm loader, which is a really common way to use TypeScript. It's not a small niche use case that just happens to affect me. Please come to the TypeScript discord and hang out for a few days, I guarantee you'll see someone coming in there to complain that ts-node doesn't work on node 20.

My suggestion of a rollback is not a cavalier suggestion or coming from any sort of animosity or judgement. This is normal OSS software dev stuff. It's an experimental API, and sometimes the result of an experiment is "oops, need to back up and try that a different way". There's no shame in it, and it doesn't reflect poorly on anyone who worked on the change.

[isaacs]

Ah! It looks like there is a way to make this work, using the globalPreload.

So then the solution appears to be to set up require.extensions in that method, instead of just in the loader itself? That definitely changes my opinion on rolling back, but I do think it could be more obvious (assuming there isn't some reason why this is a terrible idea, in which case, I would very much appreciate if someone could please set me straight!)

TypeStrong/ts-node#2009

[GeoffreyBooth]

I would be happy to. Who is this team? What’s the best way to engage with them?

This team is the folks at @nodejs/loaders, and they’re on this thread and on the OpenJS Slack at #nodejs-loaders.

This is a blocker for a project I currently care a lot about, so I’m motivated to see it addressed. I am willing and able to dive in to solve the problem.

Great! There’s already unanimous agreement that we want a load hook with type: 'commonjs' to work. If you can make that happen, without any unrelated changes (like without undoing the off-thread work and related refactors, for example) then I would be happy to land it. As it says in https://github.com/nodejs/loaders#milestone-2-stability, the most relevant links to review are #34753 (comment) and https://github.com/nodejs/loaders-test/blob/835506a638c6002c1b2d42ab7137db3e7eda53fa/coffeescript-loader/loader.js#L45-L50 and of course #44710; the PR in that last link probably includes the files you’ll need to change, or at least siblings of the files you’ll need to change, as we needed to refactor most of the ESM loader to make off-thread happen. That’s where to look. The last time we looked into it was about a year ago, before the off-thread work, and the thinking then was that the “syncification” we’d get by adding the loaders thread might provide a way to make this possible now.

Note that it’s also on our to-do list to remove globalPreload, so I wouldn’t start relying on it.

[isaacs]

Note that it’s also on our to-do list to remove globalPreload, so I wouldn’t start relying on it.

As long as it's not removed before the ability to transpile cjs in the loader arrives, or removed in such a way that its presence causes problems, I think that's fine. Such is life with experimental features, crufty workarounds sometime have to stay for a while.

[guybedford]

globalPreload was explicitly designed to continue supporting this use case of main thread instrumentation, CJS loader hooks was in mind during discussions on it. I can second that it shouldn't be removed without a suitable alternative for this use case.

[Qard]

"Breaking changes", in the sense of changes that require consumers to adjust their consuming code, as in a break from the past API to a new thing, yes. "Just broken, without any added benefit," no. That's not what "breaking changes" means, and "experimental" is not a license to ship a broken program.

I would be careful about defining experimental that way. When something is marked experimental there should be no promise it doesn't just disappear at any time or change completely. It makes it very difficult to iterate and evolve an experimental feature to something that works well if we don't have the flexibility to make incompatible changes.

That having been said, it is a difficult case when there is no non-experimental alternative. I think we need to be clearer when features like this are introduced that they are not ready for production use and that users should expect that it could change at any time until we state otherwise. Relying on loaders while they are still very much experimental was a mistake. Likely it's partly our own fault for not being clear enough about stability expectations. Also probably should have stuck to only have --experimental-loader until loaders were actually stable. 🤔

I've had discussions before that we should probably have a stability index step between "experimental" and "stable" to indicate that something is not at risk of change at any time but not proven enough to call "stable" yet. Possibly worth bringing up again. 🤔

[GeoffreyBooth]

probably should have stuck to only have --experimental-loader until loaders were actually stable. 🤔

Support for --loader (without experimental in the flag name) shipped by accident years ago and went unnoticed for over a year, after which time it felt too late to remove it. But we've never dropped the printed experimental warning. No one should be unaware that this is an experimental API.

I’ve had discussions before that we should probably have a stability index step between “experimental” and “stable” to indicate that something is not at risk of change at any time but not proven enough to call “stable” yet. Possibly worth bringing up again. 🤔

That actually landed: #46100. We just haven’t gotten around to updating all the various sections of the docs to define currently experimental features according to the new scale. If you have time to update the sections you know well, PRs would be greatly appreciated.

[isaacs]

@Qard

Relying on loaders while they are still very much experimental was a mistake

Ok, but... this is not really a case where a platform can reasonably take a hard line stance in general.

Either no one ever uses the experimental features "for real", as you suggest here, and the node never gets proper play testing (so why bother shipping them in the first place), or some significant projects do put them through their paces, which requires actually using them in some load-bearing ways, in which case there's bound to be grumbling if they're broken capriciously or without alternatives. (And if capricious breakage happens frequently, no one will trust that it's safe to even try the experimental feature for anything other than toy use cases, and you're back to not getting the play testing the project needs in order to ship something really great.)

The best approach is very much a case-by-case "it depends". If no one is using the feature, ok, maybe the experiment showed it's not useful, so go ahead and scrap it break it whatever. If people are using it, well, it's marked experimental, so yes you can probably get away with making more aggressive changes than something marked "stable", and while those changes may involve significant changes downstream, they shouldn't make it impossible to support the use cases people are using them for. In that case, there has to be a path forward, and some way to maintain backwards compatibility, even if involves a bunch of crufty version detection.

I think everyone using loaders in a serious way right now fully understands that they're experimental, and likely to change. We've all got piles of glue code to make our libs work in the various versions of the loaders API, that's fine.

My initial reaction to the off-thread loaders was based on not realizing that it would be possible to accomplish on-thread behaviors using globalPreload. Which, I still maintain would have been a bad call, but turns out I was wrong and that wasn't the case. I think maybe that workaround could be made more obvious in the docs and changelogs, and it would have been nice if it'd shipped first in an odd-number release series to catch that disruption in an unstable version, but of course sometimes it's hard to predict what's going to be an issue ahead of releasing it into the wild. And since it's a breaking change, I can see how shipping it in v19 might've been worse, and waiting until v21 might feel like it's too much of a delay. Hard stuff is hard.

[aduh95]

Should we close this issue now that ESM loader can load CJS (allowing to shortcutting the CJS loader (almost) entirely), and register (allowing to register loaders AND monkey-patching CJS loader with only one CLI flag)?

[GeoffreyBooth]

Should we close this issue now that ESM loader can load CJS (allowing to shortcutting the CJS loader (almost) entirely), and register (allowing to register loaders AND monkey-patching CJS loader with only one CLI flag)?

I think it’s safe to close this, and a new issue can be opened for whatever problems there are with the new approach. As in, I don’t think we’ve completely satisfied the original ask because Fastify needs an update to use the new APIs, but I would assume that once it’s updated then the use case should be satisfied; and if it’s not, a new issue should be opened.