-
Notifications
You must be signed in to change notification settings - Fork 30k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
loader: refactor loader #16874
loader: refactor loader #16874
Conversation
one thing i wonder, as |
lib/internal/loader/Loader.js
Outdated
const debug = require('util').debuglog('esm'); | ||
|
||
const kDynamicTranslate = Symbol('dynamicTranslate'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this a symbol if it is essentially a publicly mutable property?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I want to force people to use hook, in case we have any internal structure changes in the future, and I should probably do the same with resolve
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
They can get this via Object.getOwnPropertySymbols
, I don't think this is private.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@bmeck i think i'll use a closure but if you have a better idea lemme know. technically the loader spec wants them to be as symbols (which is why i did Loader.resolve) but i'm not too sure about that
lib/internal/loader/Loader.js
Outdated
this.resolver = resolve.bind(null); | ||
this.dynamicInstantiate = dynamicInstantiate; | ||
this.base = base; | ||
this.registry = new Registry(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please use current standard terminology: https://html.spec.whatwg.org/multipage/webappapis.html#concept-settings-object-module-map
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree, the code should keep using that terminology. It sucks that that really makes it less comprehensible, though … so I would really not mind any kind of comments to explain what’s happening
Please use the standard terminology from published specs like module map. The author of the loader spec recommends excluding it from implementations for now. |
test/es-module/test-esm-snapshot.mjs
Outdated
import one from './esm-snapshot'; | ||
import assert from 'assert'; | ||
|
||
assert.strictEqual(one, 1); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this file is done to test timing of snapshotting. It should not be removed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so this test is checking if mutating require.cache affects the imported modules? Seeing as module._load is used wouldn't this be the expected behavior?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
wouldn't this be the expected behavior?
I don't know what "this" is.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sorry about that, i meant wouldn't
- importing the mutator
- importing the mutatee (i hope thats a word)
- mutatee being mutated
be the expected behavior
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we walk thought the code the following happens:
require('./esm-snapshot-mutator')
- Module Record is formed for
./esm-snapshot-mutator
require('./esm-snapshot')
- Module Record is formed for
./esm-snapshot
- Evaluation of
./esm-snapshot
ends,module.exports === 1
- Snapshot takes effect, Module Record saves as
default = 1
- Module Record is formed for
- Evaluation of
./esm-snapshot-mutator
ends
- Module Record is formed for
The snapshot is taken before ./esm-snapshot-mutator
changes the entry of require.cache
.
This snapshot ensures the Module Namespace values of ./esm-snapshot
is the same no matter when it is imported.
lib/loader.js
Outdated
|
||
const Loader = require('internal/loader/Loader'); | ||
|
||
module.exports = new Loader(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not comfortable exposing the loader when hooks are not even finished. Please remove this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if its marked as experimental is that ok?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd still lean on no. I'm not entirely sure why you want it exposed if you want people to use hooks to mutate it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@bmeck Node’s lack of APIs for its internals has always been a huge pain point. We should definitely expose the Loader
at some point.
if you want people to use hooks to mutate it.
That makes sense if you think of the loader as Node’s way of loading its users’ modules from disk. Node definitely should have some kind of API for people to create and build modules. Think of it like the vm
module, just more complicated because modules are more complicated than scripts.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Btw, this is going to conflict with @JacksonTian’s https://npm.im/loader
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@devsnek I'd need to do a code review of it since I didn't design it for public exposure.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@devsnek is there a use case you are trying to get by doing this? I keep asking the question but don't understand right now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would like to expose it to allow for more control over node's loader, which will be a very important system moving forward. In the same way that require can be messed with, I think that loader should be able to be messed with. I agree that there should be control over what stuff can be messed with, but the loader having a public api is, in my opinion, a must have moving forward. I think that the hooks idea is good as a controlled system (and should remain in), but that it should also be possible to make your own loaders, change them up, etc. More to this point is why i suggested that at least exposing the class by default while allowing access to the instance with a flag would be a good choice to keep the system save and extensible at the same time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would need to understand various aspects of this. I see you want it, but don't see why except an opinion of "I think that loader should be able to be messed with". Various parts of mutating require like NODE_PATH
and require.extensions
were deprecated and I don't feel that saying require
is mutable so the future should be mutable is a strong position. Exposing individual use cases like hooking into how specifier resolution works is very different from creating multiple disconnected loaders.
In particular, this exposure is problematic since the Module Map is not shared. I would not expose this without a clearer reasoning to need/want multiple completely disconnected loaders.
I also feel like the Realms API should be looked at to ensure we don't have problems if/when it arrives. The people who worked on the WHATWG Loader API are involved in that. It doesn't look to be exposing a Loader like object from the last few TC39 meetings on this topic.
If you can explain the concrete use case of having multiple loaders and explain how it does not suffer from the problems of mutation in the same way that require
has that seems fine. Right now this seems too rushed and hasn't looked at compatibility or scope of what exposing it means.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"Messing with" the loader won't just have effects for you and your app, it could very very easily leak out and have effects on the ecosystem.
Whether this is a good idea ever is a separate discussion, but allowing this kind of experimentation prior to everything being stable and finished is, imo, a very dangerous and bad idea.
lib/loader.js
Outdated
|
||
const Loader = require('internal/loader/Loader'); | ||
|
||
module.exports = new Loader(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
#16874 (comment) remains, commenting since default github UI is hiding it
lib/internal/loader/Loader.js
Outdated
} | ||
// A closure is used here because we really really really don't want users | ||
// touching the hooks without using `hook()`. | ||
function Loader(base = getURLStringForCwd()) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not a class?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the comment explains
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It can still be a class with the this.hook =
in the constructor
@bmeck how do |
@devsnek It has 2 parts: When
|
@bmeck i meant uh // file.js
require('fs');
// main.mjs
import 'file.js'; when running |
@devsnek it shouldn't. What is the output with |
yea i tried that earlier, got really confused:
|
Ah, it looks like node/lib/internal/loader/ModuleRequest.js Line 30 in d597317
I don't see |
@bmeck on my latest commit i added |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not really a review, just a quick glance.
lib/internal/module.js
Outdated
@@ -86,7 +86,7 @@ const builtinLibs = [ | |||
'assert', 'async_hooks', 'buffer', 'child_process', 'cluster', 'crypto', | |||
'dgram', 'dns', 'domain', 'events', 'fs', 'http', 'http2', 'https', 'net', | |||
'os', 'path', 'perf_hooks', 'punycode', 'querystring', 'readline', 'repl', | |||
'stream', 'string_decoder', 'tls', 'tty', 'url', 'util', 'v8', 'vm', 'zlib' | |||
'stream', 'string_decoder', 'tls', 'tty', 'url', 'util', 'v8', 'vm', 'zlib', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unrelated change?
test/es-module/test-esm-snapshot.mjs
Outdated
@@ -1,5 +1,6 @@ | |||
// Flags: --experimental-modules | |||
/* eslint-disable required-modules */ | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Likewise.
@@ -5,7 +5,7 @@ const debug = require('util').debuglog('esm'); | |||
const ArrayJoin = Function.call.bind(Array.prototype.join); | |||
const ArrayMap = Function.call.bind(Array.prototype.map); | |||
|
|||
const createDynamicModule = (exports, url = '', evaluate) => { | |||
function createDynamicModule(exports, url = '', evaluate) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please change this back to an arrow function, it lacks a .prototype
and protects against new
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
will do, also did you see #16874 (comment)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I did, but this PR is a lot of changes which makes me a bit slow to re-review so quickly
as far as i can tell at this point, module_wrap wants to have the full module at instantiate time, not just the dependencies. is this safe to change? technically if the module has no deps, it shouldn't need the full module until evaluation time. |
@devsnek InnerModuleInstantiation requires the full Module since it does instantiate the hoistable declarations using ModuleDeclarationEnvironmentSetup |
I think |
@bmeck but if there aren't any hoistable declarations there's no need to have the full module right? |
@devsnek you still need it whenever a parent is instantiating. The whole graph Instantiates prior to any evaluation. Not sure I understand |
being able to defer translation until |
lib/internal/loader/Loader.js
Outdated
// Use .bind() to avoid giving access to the Loader instance when it is | ||
// called as this.resolver(...); | ||
this._resolve = resolve.bind(null); | ||
this._dynamicTranslate = dynamicTranslate; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
bind(null) here too
Note this still needs a documentation change that the |
I missed the |
It's also smooshing case though as it's not "Commonjs". Using |
This is not landing cleanly on v9.x, would someone be willing to open a backport? |
I can do that when I get home in a few hours. i should open a pr targeting the 9.x branch right? |
@devsnek nevermind, I got it to land after another backport landed! |
PR-URL: #16874 Reviewed-By: Guy Bedford <guybedford@gmail.com> Reviewed-By: Bradley Farias <bradley.meck@gmail.com>
Should this be backported to |
this sits on top of #18085, so i'm not sure |
If this is backported, it should be backported along with #18596. |
This restores a broken and erroneously removed error, which was accidentially renamed to ERR_MISSING_DYNAMIC_INTSTANTIATE_HOOK (notice the "INTST" vs "INST") in 921fb84 (PR #16874) and then had documentation and implementation removed under the old name in 6e1c25c (PR #18857), as it appeared unused. This error code never worked or was documented under the mistyped name ERR_MISSING_DYNAMIC_INTSTANTIATE_HOOK, so renaming it back to ERR_MISSING_DYNAMIC_INSTANTIATE_HOOK is a semver-patch fix. Refs: #21440 Refs: #21470 Refs: #16874 Refs: #18857
This restores a broken and erroneously removed error, which was accidentially renamed to ERR_MISSING_DYNAMIC_INTSTANTIATE_HOOK (notice the "INTST" vs "INST") in 921fb84 (PR nodejs#16874) and then had documentation and implementation removed under the old name in 6e1c25c (PR nodejs#18857), as it appeared unused. This error code never worked or was documented under the mistyped name ERR_MISSING_DYNAMIC_INTSTANTIATE_HOOK, so renaming it back to ERR_MISSING_DYNAMIC_INSTANTIATE_HOOK is a semver-patch fix. Refs: nodejs#21440 Refs: nodejs#21470 Refs: nodejs#16874 Refs: nodejs#18857
#21493 also should be included with any backport. |
This restores a broken and erroneously removed error, which was accidentially renamed to ERR_MISSING_DYNAMIC_INTSTANTIATE_HOOK (notice the "INTST" vs "INST") in 921fb84 (PR #16874) and then had documentation and implementation removed under the old name in 6e1c25c (PR #18857), as it appeared unused. This error code never worked or was documented under the mistyped name ERR_MISSING_DYNAMIC_INTSTANTIATE_HOOK, so renaming it back to ERR_MISSING_DYNAMIC_INSTANTIATE_HOOK is a semver-patch fix. Refs: #21440 Refs: #21470 Refs: #16874 Refs: #18857 PR-URL: #21493 Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com> Reviewed-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com> Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Ron Korving <ron@ronkorving.nl> Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
This restores a broken and erroneously removed error, which was accidentially renamed to ERR_MISSING_DYNAMIC_INTSTANTIATE_HOOK (notice the "INTST" vs "INST") in 921fb84 (PR #16874) and then had documentation and implementation removed under the old name in 6e1c25c (PR #18857), as it appeared unused. This error code never worked or was documented under the mistyped name ERR_MISSING_DYNAMIC_INTSTANTIATE_HOOK, so renaming it back to ERR_MISSING_DYNAMIC_INSTANTIATE_HOOK is a semver-patch fix. Refs: #21440 Refs: #21470 Refs: #16874 Refs: #18857 PR-URL: #21493 Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com> Reviewed-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com> Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Ron Korving <ron@ronkorving.nl> Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
The loader was refactored to use some better terms and methods, mostly
inspired by the whatwg loader spec. Namely there is now
translators
instead of
loaders
andregistry
instead ofmoduleMap
. The loaderis also exposed as a builtin, allowing users to make their own loaders
and hook the main loader.
I also found a test that wanted to check for the values of exports being unchanged, but I think thats not really correct, and only succeeded as an edge case, and since it was failing after the refactor I removed it. If this was erroneous please let me know.
I'm going to avoid documenting the whole loader until this gets some backing.
cc @bmeck @guybedford @addaleax
Checklist
make -j4 test
(UNIX), orvcbuild test
(Windows) passesAffected core subsystem(s)
loader