module: detect ESM syntax by trying to recompile as SourceTextModule #52413

joyeecheung · 2024-04-07T22:02:34Z

Instead of using an async function wrapper, just try compiling code with unknown module format as SourceTextModule when it cannot be compiled as CJS and the error message indicates that it's worth a retry. If it can be parsed as SourceTextModule then it's considered ESM.

Also, move shouldRetryAsESM() to C++ completely so that we can reuse it in the CJS module loader for require(esm).

Drive-by: move methods that don't belong to ContextifyContext out as static methods and move GetHostDefinedOptions to ModuleWrap.

nodejs-github-bot · 2024-04-07T22:02:39Z

Review requested:

@nodejs/loaders
@nodejs/vm

guybedford · 2024-04-07T22:16:13Z

This seems like it works. Can we think of any cases where a module would not use top-level await, import or export or import.meta syntax, and does not lexically define any of the CJS variables, where it would parse as ESM and not a script function body? I can't think of anything but it would be great if we could fully confirm the edge cases.

joyeecheung · 2024-04-07T23:09:34Z

I think no matter what edge cases there are, it's pretty unlikely that reparsing it with a weird async function wrapper (what the main branch currently does) can be any more correct than reparsing as SourceTextModule....I'd prefer to just go ahead with this to unblock #52047

lib/internal/modules/esm/get_format.js

nodejs-github-bot · 2024-04-07T23:54:45Z

CI: https://ci.nodejs.org/job/node-test-pull-request/58196/

GeoffreyBooth · 2024-04-08T00:53:51Z

There are two reasons for the current behavior:

If we reparse as ESM and that fails too, presumably the initial parse failed for a reason other than ESM syntax and so we want to throw that error, not whatever error is thrown by attempting to parse as ESM.
There's no need to reparse as ESM for any CommonJS parse error, so for efficiency we want to do the reparse only when it might make a difference.

I'm on my phone so I can't tell if these have been addressed, but if so, then sure let's change the algorithm (and update the docs accordingly). I think the first point has an existing test covering it, but if not then we should add one.

joyeecheung · 2024-04-08T14:34:35Z

If we reparse as ESM and that fails too, presumably the initial parse failed for a reason other than ESM syntax and so we want to throw that error, not whatever error is thrown by attempting to parse as ESM.

That's not changed. The error getting rethrown from catch block in executeUserEntryPoint() is still the error from the CJS loader. Reparse errors are still swallowed, the only difference is that previously it was re-parsed as wrapped async function, now it's reparsed as ESM. That's covered by test/fixtures/es-modules/package-without-type/commonjs-wrapper-variables.js which still passes as you can see in the CI.

There's no need to reparse as ESM for any CommonJS parse error, so for efficiency we want to do the reparse only when it might make a difference.

That's not changed either. It is still only reparsed when the error is in the throws_only_in_cjs_error_messages set.

joyeecheung · 2024-04-08T14:39:35Z

then sure let's change the algorithm (and update the docs accordingly).

I don't think the algorithm as documented changes in anyway as it does not mention specifics about how the code is reparsed (would be somewhat funny to mention that async wrapper too). The doc only lists errors used to determine whether the code should be retried as ESM, those aren't changed by this PR. We are still using the same set to determine syntax errors that definitely indicate ESM syntax and don't need a reparse, and the same set to determine syntax errors that warrant a reparse to double-check.

lib/internal/modules/run_main.js

nodejs-github-bot · 2024-04-11T12:42:52Z

CI: https://ci.nodejs.org/job/node-test-pull-request/58280/

nodejs-github-bot · 2024-04-11T22:44:51Z

CI: https://ci.nodejs.org/job/node-test-pull-request/58292/

GeoffreyBooth · 2024-04-11T23:14:06Z

lib/internal/modules/run_main.js

- const { enrichCJSError } = require('internal/modules/esm/translators');
- enrichCJSError(error, source, resolvedMain);


How is removing these lines not causing any tests to break? There are tests that check the output of this “enriched” error that I’d think should be failing without these lines.

Those only check part of the sentence because the rest keeps changing (typically it's not a good idea to check all of the message, either). The error is now "enriched" in C++.

Actually, I was talking about my followup. In this branch, this part is already just unreachable, even before this patch, because if !retryAsESM, then the code doesn't contains ESM syntax, then the error won't be "enriched" by enrichCJSError() anyway (i.e. skipping inside enrichCJSError() already).

lib/internal/modules/run_main.js

GeoffreyBooth · 2024-04-12T04:49:14Z

src/node_contextify.cc

+ try_catch.ReThrow();
+ return;
+ }
+ }


Does this consolidation mean that we no longer need the try/catch at all? In other words, we could revert #52093 to go back to the previous approach of calling containsModuleSyntax, and then running the CJS entry with the CJS loader, and the performance cost will be negligible because the parse gets cached in V8?

Yes, that's what my follow up does https://github.com/joyeecheung/node/commits/esm-detect-cpp/ which also implements detection support for require(esm): if it's entry point, and the detection flag is enabled, and it looks like ESM, use the path that allow TLA, and ignore the exports because uh no one needs the module.exports of the entry point..at least for something that's already ESM because that would not have been possible anyway? If it's not entry point, and the require flag is enabled, or detection flag is enabled and it looks like ESM, do synchronous require esm and add module.exports. Though I don't think it's a good idea to merge that into this PR, this is a refactoring, that one implement things

if it’s entry point, and the detection flag is enabled, and it looks like ESM, use the path that allow TLA, and ignore the exports because uh no one needs the module.exports of the entry point

What if the entry point gets required or imported a second time? Like node entry.js imports foo.js which imports entry.js? The second import would need the exports from entry.js.

Can we land your follow-up and unflag detect-module while require(esm) is still flagged? Or is require(esm) possibly close enough to being ready to unflag that we could just do them together?

The second import would need the exports from entry.js.

If entry.js is ESM, then that's the namespace, not module.exports, and we'll return the namespace of the cached module in the ESM loader, because import entry.js is not handled by CJS loader when the module job is already cached.

If it's require entry.js again, then we can error or warn that it's the entry point, or return the namespace if it's fully evaluated. But that's talking about require(esm) edge cases in a follow-up now, pretty off topic for this PR.

Can we land your follow-up and unflag detect-module while require(esm) is still flagged?

I don't know for sure, locally my followup does pass test/parallel/test-debugger-exceptions.js even when detect-module is unflagged, though, if you think that's the last blocker for unflagging detection. require(esm) will need to be flagged for at least 22, blockers are __esModule interop and customization hooks support, and maybe there are more as we get more feedback from 22. In general I'm not too comfortable with just unflagging an experimental feature that really needs some battle testing from users at least.

In general I'm not too comfortable with just unflagging an experimental feature that really needs some battle testing from users at least.

Especially for a feature like this one. As soon as it can be used without a flag, people will start to publish packages that depend on it. Then we will be blamed if we break them.

src/node_contextify.cc

nodejs-github-bot · 2024-04-12T16:11:11Z

CI: https://ci.nodejs.org/job/node-test-pull-request/58326/

nodejs-github-bot · 2024-04-14T01:18:42Z

CI: https://ci.nodejs.org/job/node-test-pull-request/58353/

JakobJingleheimer · 2024-04-14T09:47:07Z

lib/internal/modules/run_main.js

- // In case the entry point is a large file, such as a bundle,
- // ensure no further references can prevent it being garbage-collected.
- cjsLoader.entryPointSource = undefined;
+ if (error != null && ObjectGetPrototypeOf(error) === SyntaxErrorPrototype) {


Potential follow-up: Do we control the error being thrown? If so, I'm wondering if we should create a sub-class like MismatchedSyntaxError 🤔

The follow up is just don't do the check in JS, and check right after parsing in C++ :)

lib/internal/modules/run_main.js

nodejs-github-bot · 2024-04-14T14:11:05Z

CI: https://ci.nodejs.org/job/node-test-pull-request/58371/

nodejs-github-bot · 2024-04-15T15:57:49Z

CI: https://ci.nodejs.org/job/node-test-pull-request/58402/

nodejs-github-bot · 2024-04-15T19:38:33Z

CI: https://ci.nodejs.org/job/node-test-pull-request/58410/

nodejs-github-bot · 2024-04-16T13:59:49Z

CI: https://ci.nodejs.org/job/node-test-pull-request/58433/

nodejs-github-bot · 2024-04-17T00:24:30Z

CI: https://ci.nodejs.org/job/node-test-pull-request/58447/

Instead of using an async function wrapper, just try compiling code with unknown module format as SourceTextModule when it cannot be compiled as CJS and the error message indicates that it's worth a retry. If it can be parsed as SourceTextModule then it's considered ESM. Also, move shouldRetryAsESM() to C++ completely so that we can reuse it in the CJS module loader for require(esm). Drive-by: move methods that don't belong to ContextifyContext out as static methods and move GetHostDefinedOptions to ModuleWrap.

nodejs-github-bot · 2024-04-18T22:10:06Z

CI: https://ci.nodejs.org/job/node-test-pull-request/58507/

nodejs-github-bot · 2024-04-18T23:49:24Z

CI: https://ci.nodejs.org/job/node-test-pull-request/58508/

nodejs-github-bot · 2024-04-19T14:29:09Z

Landed in 651fa04

Instead of using an async function wrapper, just try compiling code with unknown module format as SourceTextModule when it cannot be compiled as CJS and the error message indicates that it's worth a retry. If it can be parsed as SourceTextModule then it's considered ESM. Also, move shouldRetryAsESM() to C++ completely so that we can reuse it in the CJS module loader for require(esm). Drive-by: move methods that don't belong to ContextifyContext out as static methods and move GetHostDefinedOptions to ModuleWrap. PR-URL: #52413 Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com> Reviewed-By: Jacob Smith <jacob@frende.me>

nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels Apr 7, 2024

joyeecheung mentioned this pull request Apr 7, 2024

module: support ESM detection in the CJS loader #52047

Merged

anonrig reviewed Apr 7, 2024

View reviewed changes

lib/internal/modules/esm/get_format.js Show resolved Hide resolved

joyeecheung added the request-ci Add this label to start a Jenkins CI on a PR. label Apr 7, 2024

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Apr 7, 2024

aduh95 reviewed Apr 8, 2024

View reviewed changes

lib/internal/modules/run_main.js Outdated Show resolved Hide resolved

lib/internal/modules/run_main.js Outdated Show resolved Hide resolved

joyeecheung mentioned this pull request Apr 8, 2024

module: improve detection try catch #52242

Closed

joyeecheung force-pushed the refactor-esm-detection branch from 179dabd to dc9f175 Compare April 11, 2024 12:35

joyeecheung added the request-ci Add this label to start a Jenkins CI on a PR. label Apr 11, 2024

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Apr 11, 2024

GeoffreyBooth reviewed Apr 12, 2024

View reviewed changes

joyeecheung force-pushed the refactor-esm-detection branch from 1b571e1 to 6eec04e Compare April 12, 2024 15:57

joyeecheung added the request-ci Add this label to start a Jenkins CI on a PR. label Apr 12, 2024

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Apr 12, 2024

GeoffreyBooth approved these changes Apr 12, 2024

View reviewed changes

JakobJingleheimer approved these changes Apr 14, 2024

View reviewed changes

joyeecheung added the request-ci Add this label to start a Jenkins CI on a PR. label Apr 14, 2024

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Apr 14, 2024

joyeecheung added the request-ci Add this label to start a Jenkins CI on a PR. label Apr 17, 2024

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Apr 17, 2024

joyeecheung force-pushed the refactor-esm-detection branch from 425d8ee to 14afc5a Compare April 18, 2024 18:36

GeoffreyBooth approved these changes Apr 18, 2024

View reviewed changes

GeoffreyBooth added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Apr 19, 2024

joyeecheung mentioned this pull request Apr 19, 2024

module: address coverity warning of memory leak #52586

Closed

joyeecheung added the commit-queue Add this label to land a pull request using GitHub Actions. label Apr 19, 2024

nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Apr 19, 2024

nodejs-github-bot merged commit 651fa04 into nodejs:main Apr 19, 2024
41 checks passed

aduh95 mentioned this pull request Apr 30, 2024

2024-05-02, Version 22.1.0 (Current) #52768

Merged

marco-ippolito added the backport-blocked-v20.x PRs that should land on the v20.x-staging branch but are blocked by another PR's pending backport. label Jul 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

module: detect ESM syntax by trying to recompile as SourceTextModule #52413

module: detect ESM syntax by trying to recompile as SourceTextModule #52413

joyeecheung commented Apr 7, 2024

nodejs-github-bot commented Apr 7, 2024

guybedford commented Apr 7, 2024

joyeecheung commented Apr 7, 2024 •

edited

Loading

nodejs-github-bot commented Apr 7, 2024

GeoffreyBooth commented Apr 8, 2024

joyeecheung commented Apr 8, 2024 •

edited

Loading

joyeecheung commented Apr 8, 2024 •

edited

Loading

nodejs-github-bot commented Apr 11, 2024

nodejs-github-bot commented Apr 11, 2024

GeoffreyBooth Apr 11, 2024

joyeecheung Apr 12, 2024

joyeecheung Apr 12, 2024 •

edited

Loading

GeoffreyBooth Apr 12, 2024

joyeecheung Apr 12, 2024 •

edited

Loading

GeoffreyBooth Apr 12, 2024

joyeecheung Apr 12, 2024 •

edited

Loading

joyeecheung Apr 12, 2024 •

edited

Loading

targos Apr 12, 2024

nodejs-github-bot commented Apr 12, 2024

nodejs-github-bot commented Apr 14, 2024

JakobJingleheimer Apr 14, 2024

joyeecheung Apr 14, 2024 •

edited

Loading

nodejs-github-bot commented Apr 14, 2024

nodejs-github-bot commented Apr 15, 2024

nodejs-github-bot commented Apr 15, 2024

nodejs-github-bot commented Apr 16, 2024

nodejs-github-bot commented Apr 17, 2024

nodejs-github-bot commented Apr 18, 2024

nodejs-github-bot commented Apr 18, 2024

nodejs-github-bot commented Apr 19, 2024

		const { enrichCJSError } = require('internal/modules/esm/translators');
		enrichCJSError(error, source, resolvedMain);

module: detect ESM syntax by trying to recompile as SourceTextModule #52413

module: detect ESM syntax by trying to recompile as SourceTextModule #52413

Conversation

joyeecheung commented Apr 7, 2024

nodejs-github-bot commented Apr 7, 2024

guybedford commented Apr 7, 2024

joyeecheung commented Apr 7, 2024 • edited Loading

nodejs-github-bot commented Apr 7, 2024

GeoffreyBooth commented Apr 8, 2024

joyeecheung commented Apr 8, 2024 • edited Loading

joyeecheung commented Apr 8, 2024 • edited Loading

nodejs-github-bot commented Apr 11, 2024

nodejs-github-bot commented Apr 11, 2024

GeoffreyBooth Apr 11, 2024

Choose a reason for hiding this comment

joyeecheung Apr 12, 2024

Choose a reason for hiding this comment

joyeecheung Apr 12, 2024 • edited Loading

Choose a reason for hiding this comment

GeoffreyBooth Apr 12, 2024

Choose a reason for hiding this comment

joyeecheung Apr 12, 2024 • edited Loading

Choose a reason for hiding this comment

GeoffreyBooth Apr 12, 2024

Choose a reason for hiding this comment

joyeecheung Apr 12, 2024 • edited Loading

Choose a reason for hiding this comment

joyeecheung Apr 12, 2024 • edited Loading

Choose a reason for hiding this comment

targos Apr 12, 2024

Choose a reason for hiding this comment

nodejs-github-bot commented Apr 12, 2024

nodejs-github-bot commented Apr 14, 2024

JakobJingleheimer Apr 14, 2024

Choose a reason for hiding this comment

joyeecheung Apr 14, 2024 • edited Loading

Choose a reason for hiding this comment

nodejs-github-bot commented Apr 14, 2024

nodejs-github-bot commented Apr 15, 2024

nodejs-github-bot commented Apr 15, 2024

nodejs-github-bot commented Apr 16, 2024

nodejs-github-bot commented Apr 17, 2024

nodejs-github-bot commented Apr 18, 2024

nodejs-github-bot commented Apr 18, 2024

nodejs-github-bot commented Apr 19, 2024

joyeecheung commented Apr 7, 2024 •

edited

Loading

joyeecheung commented Apr 8, 2024 •

edited

Loading

joyeecheung commented Apr 8, 2024 •

edited

Loading

joyeecheung Apr 12, 2024 •

edited

Loading

joyeecheung Apr 12, 2024 •

edited

Loading

joyeecheung Apr 12, 2024 •

edited

Loading

joyeecheung Apr 12, 2024 •

edited

Loading

joyeecheung Apr 14, 2024 •

edited

Loading