-
Notifications
You must be signed in to change notification settings - Fork 30k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
implement module.registerHooks() to run synchronous module customization hooks in thread #55698
Conversation
Review requested:
|
d96846a
to
decfb68
Compare
decfb68
to
7778156
Compare
Added a lot of tests. This should now be ready for review. |
This comment was marked as outdated.
This comment was marked as outdated.
7778156
to
9d7b53a
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #55698 +/- ##
==========================================
+ Coverage 88.50% 88.54% +0.03%
==========================================
Files 656 657 +1
Lines 189261 189858 +597
Branches 36346 36451 +105
==========================================
+ Hits 167508 168101 +593
- Misses 14969 14976 +7
+ Partials 6784 6781 -3
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We've discussed this in the past and my previous feedback still stands that overall I'm very much for this direction, with the caveat that we just need to ensure we guide hooks authors to patterns that work well for ensuring consistent behaviours including for environments that use workers.
Specifically it would be great if we can recommend the --import
top-level hook registration pattern quite strongly, as that naturally works with workers and it's worth calling out that this does that explicitly when documenting registerHooks
.
I do have one feature request I'd love to see here though - a one-way function to disable hooks for security module.lockHooks()
, with no arguments, that disallows further register
or registerHooks
calls.
Something to this effect would be a huge help to ensure runtime hooks registered on arbitrary function calls is not a thing, and we do think about the execution model as "register hooks" moving into a stage of "hooks are registered and can't be registered", rather than thinking of hooks as being able to happen at any time.
That seems out of scope of this PR? If it was fine not having it for |
Having a hook lockdown phase is not a requirement for this PR by any means, but would be a nice to have to clarify the early hook registration model sooner rather than later. |
I feel that should be something that's up to the discretion of the hook author, there can surely be use cases where one wants to avoid inheritance in third-party child processes/workers, or just to avoid infinite recursion in case the code run in the loader forks itself, though we could mention it in the documentation for sure. |
Right, so it's our job to educate hook authors how to write hooks to support multi threading (or not) and how to avoid recursive application etc. etc. With regards to hook "lockdown" I think this is something that the application owner would care about - how do I know that an arbitrary npm package import isn't going to be doing hooks without me knowing about it? How do I know I'm not depending on a package that's automatically adding a "jsx" hook and changing the way my local code runs? A lockdown feature is for the application code owner's benefit. Ideally lockdown by default would be great too to ensure we don't enter a Node.js future where hooking the loader is common for library dependencies. |
While I agree better documentation should exist, I think we shouldn't really dive into too much details in the API documentation. It should either be in the learn section in the website, or just have dedicated repository for tutorials + examples like https://github.com/nodejs/node-addon-examples and the new https://github.com/nodejs/package-examples that we are spinning (I also left comment in this PR that I think the example loaders should be moved in its own repo, and refrained from adding too many complementary examples for the new API, I also noticed that some existing examples in the documentation didn't even work, but it felt out of scope to fix them here and IMO they should really just be tested properly in a dedicated repo). |
@joyeecheung to give an example, I was not aware |
Yes, though not all valuable information belongs in the API documentation ;) One can argue that https://nodejs.github.io/node-addon-examples/ is full of valuable information, but it's better to have a dedicated tutorial on a complex topic like this, instead of trying to elaborate on it in the API documentation. I think we can make do with one or two sentence about |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great work! I’m excited to see this land.
doc/api/module.md
Outdated
@@ -693,8 +703,24 @@ register('./my-hooks.mjs', { | |||
}); | |||
``` | |||
|
|||
### In-thread Synchronous Customization Hooks |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I feel like we should list these first, since we intend these to be the way forward.
I think the docs should also be a bit direct and state that we intend to remove register
once there is a migration path to registerHooks
. With register
still being categorized as “release candidate”, I wouldn’t want someone seeing register
as more mature and more stable than registerHooks
and therefore a safer choice.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wouldn’t want someone seeing register as more mature and more stable than registerHooks and therefore a safer choice.
I think that is an accurate description for now, considering module.registerHooks()
is not yet even released, and module.register()
has already been used by several projects. The future migration path should be a follow up after we actually get some feedback from the user land. This is a semver-minor PR and we should keep it so. The problem of the off-thread hooks came from dictating a path and gearing too many architectural decisions towards it and closing all other doors before sufficient real-world feedback was collected, and I don't think we should make the same mistake again just because we believe "this time, we got it right!" (I hope we are, but this isn't up to us).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you don’t want to include anything about “we consider this to be the way forward” just yet, fine, but I still think that the new hooks should go first; and the explanation about how the hooks compare with each other should live with the async hooks rather than with the sync hooks. That way when we remove the async hooks, the comparison text is already there and goes away cleanly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I spent some time to move it and it took way to much effort to do an overhaul of the texts to describe the sync hooks first, because the existing texts always assume they are talking about the async ones and need wording changes everywhere. I don't think this needs to be done in the initial PR that's mostly just implementing things. Can you open a follow-up PR to do the rewrite?
doc/api/module.md
Outdated
`module.registerHooks()` is a lighter weight alternative to `module.register()` | ||
which takes hook functions directly and run them in the same thread where the | ||
modules are loaded. The hook functions are synchronous, which is necessary to |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This won’t make much sense once module.register
is removed. Perhaps we should just introduce registerHooks
straightforwardly and leave any comparisons to the register
section?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think for people who are already using module.register()
and trying to figure out what the new API offers, this should be noted. We can remove it when module.register()
actually gets removed, which would take a while because the usage out there already warrants a deprecation cycle, and in that case this would also serve as what all deprecation needs in the documentation - how to migrate to the alternative of the deprecated API.
lib/internal/modules/sync_hooks.js
Outdated
// TODO(joyeecheung): we may want methods that allow disabling/enabling temporarily | ||
// which just sets the item in the array to undefined temporarily. | ||
// TODO(joyeecheung): this can be the [Symbol.dispose] implementation to pair with | ||
// `using` when the explicit resource management proposal is shipped by V8. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
May as well add the Symbol.dispose
implementation now to prepare since we have it available.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it needed for the first iteration? If not, I'd rather leave it out and just leave a TODO comment instead, lest this got derailed into another debate about whether that's the correct implementation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed this could be a follow-up. TBH, I don't really find how this could be helpful in the reality.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah someone suggested this in the proposal PR IIRC, though I think using this with using
would be a rare choice - if you want something temporarily customized in the same file, chances are you are going to anchor it to other smaller parts of a bigger hook that just stays in the chain?
lib/internal/modules/sync_hooks.js
Outdated
// TODO(joyeecheung): we may want methods that allow disabling/enabling temporarily | ||
// which just sets the item in the array to undefined temporarily. | ||
// TODO(joyeecheung): this can be the [Symbol.dispose] implementation to pair with | ||
// `using` when the explicit resource management proposal is shipped by V8. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed this could be a follow-up. TBH, I don't really find how this could be helpful in the reality.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The readme context is looking a lot better, my main remaining suggestion would be to state the sync hooks first as the recommended path.
Furthermore, as far as I'm aware there is rough consensus to deprecate the async customization hooks. Are there plans to follow-up with a deprecation PR at all?
1. `module.register(specifier[, parentURL][, options])` which takes a module that | ||
exports asynchronous hook functions. The functions are run on a separate loader | ||
thread. | ||
2. `module.registerHooks(options)` which takes synchronous hook functions that are | ||
run directly on the thread where the module is loaded. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why don't we just put the sync hooks first?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Because all the existing texts assume they are talking about the async hooks, so to avoid doing an wording overhaul of all the existing texts, the additional explanation of the sync hooks need to come second, which then also means the bullet points should come second otherwise the ordering looks rather off. I don't think this the wording overhaul has to be done in the initial PR that's mostly just implementing things. Can you open a follow-up PR to do the rewrite?
1. Registering a file which exports a set of asynchronous hook functions, using the | ||
[`register`][] method from `node:module`, | ||
2. Registering a set of synchronous hook functions using the [`registerHooks`][] method | ||
from `node:module`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Again, might be worth putting sync hooks first as the recommended path.
doc/api/module.md
Outdated
The hooks can be registered before the application code is run by using the | ||
[`--import`][] or [`--require`][] flag: | ||
```bash | ||
node --import ./register-hooks.js ./my-app.js | ||
node --require ./register-hooks.js ./my-app.js | ||
``` | ||
```mjs | ||
// register-hooks.js | ||
// This file can only be require()-ed if it doesn't contain top-level await. | ||
// To register asynchronous, off-thread hooks, use module.register(). | ||
import { register } from 'node:module'; | ||
|
||
register('./hooks.mjs', import.meta.url); | ||
``` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Again, for this first example, it would be great to see the sync hooks workflow first, then have the alternative workflow for async hooks provided.
This comment was marked as outdated.
This comment was marked as outdated.
0f7172a
to
9529784
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Happy to leave the sync reordering to a deprecation PR for the async variant in due course.
Landed in 2960a59...1215a8b |
PR-URL: #55698 Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com> Reviewed-By: Guy Bedford <guybedford@gmail.com>
PR-URL: #55698 Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com> Reviewed-By: Guy Bedford <guybedford@gmail.com>
The
notable-change
Please suggest a text for the release notes if you'd like to include a more detailed summary, then proceed to update the PR description with the text or a link to the notable change suggested text comment. Otherwise, the commit will be placed in the Other Notable Changes section. |
PR-URL: #55698 Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com> Reviewed-By: Guy Bedford <guybedford@gmail.com>
PR-URL: #55698 Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com> Reviewed-By: Guy Bedford <guybedford@gmail.com>
Notable changes: crypto: * graduate WebCryptoAPI Ed25519 and X25519 algorithms as stable (Filip Skokan) #56142 doc: * stabilize util.styleText (Rafael Gonzaga) #56265 module: * (SEMVER-MINOR) add prefix-only modules to `module.builtinModules` (Jordan Harband) #56185 * (SEMVER-MINOR) only emit require(esm) warning under --trace-require-module (Joyee Cheung) #56194 * (SEMVER-MINOR) use synchronous hooks for preparsing in import(cjs) (Joyee Cheung) #55698 * (SEMVER-MINOR) implement module.registerHooks() (Joyee Cheung) #55698 report: * (SEMVER-MINOR) fix typos in report keys and bump the version (Yuan-Ming Hsu) #56068 sqlite: * (SEMVER-MINOR) aggregate constants in a single property (Edigleysson Silva (Edy)) #56213 src,lib: * (SEMVER-MINOR) stabilize permission model (Rafael Gonzaga) #56201 stream: * (SEMVER-MINOR) handle generator destruction from Duplex.from() (Matthieu Sieben) #55096 PR-URL: #56310
Notable changes: crypto: * graduate WebCryptoAPI Ed25519 and X25519 algorithms as stable (Filip Skokan) #56142 doc: * stabilize util.styleText (Rafael Gonzaga) #56265 module: * (SEMVER-MINOR) add prefix-only modules to `module.builtinModules` (Jordan Harband) #56185 * (SEMVER-MINOR) only emit require(esm) warning under --trace-require-module (Joyee Cheung) #56194 * (SEMVER-MINOR) use synchronous hooks for preparsing in import(cjs) (Joyee Cheung) #55698 * (SEMVER-MINOR) implement module.registerHooks() (Joyee Cheung) #55698 report: * (SEMVER-MINOR) fix typos in report keys and bump the version (Yuan-Ming Hsu) #56068 sqlite: * (SEMVER-MINOR) aggregate constants in a single property (Edigleysson Silva (Edy)) #56213 src,lib: * (SEMVER-MINOR) stabilize permission model (Rafael Gonzaga) #56201 stream: * (SEMVER-MINOR) handle generator destruction from Duplex.from() (Matthieu Sieben) #55096 PR-URL: TODO
Notable changes: crypto: * graduate WebCryptoAPI Ed25519 and X25519 algorithms as stable (Filip Skokan) #56142 doc: * stabilize util.styleText (Rafael Gonzaga) #56265 module: * (SEMVER-MINOR) add prefix-only modules to `module.builtinModules` (Jordan Harband) #56185 * (SEMVER-MINOR) only emit require(esm) warning under --trace-require-module (Joyee Cheung) #56194 * (SEMVER-MINOR) use synchronous hooks for preparsing in import(cjs) (Joyee Cheung) #55698 * (SEMVER-MINOR) implement module.registerHooks() (Joyee Cheung) #55698 report: * (SEMVER-MINOR) fix typos in report keys and bump the version (Yuan-Ming Hsu) #56068 sqlite: * (SEMVER-MINOR) aggregate constants in a single property (Edigleysson Silva (Edy)) #56213 src,lib: * (SEMVER-MINOR) stabilize permission model (Rafael Gonzaga) #56201 stream: * (SEMVER-MINOR) handle generator destruction from Duplex.from() (Matthieu Sieben) #55096 PR-URL: #56310
Notable changes: crypto: * graduate WebCryptoAPI Ed25519 and X25519 algorithms as stable (Filip Skokan) #56142 dgram: * (SEMVER-MINOR) support blocklist in udp (theanarkh) #56087 doc: * stabilize util.styleText (Rafael Gonzaga) #56265 module: * (SEMVER-MINOR) add prefix-only modules to `module.builtinModules` (Jordan Harband) #56185 * (SEMVER-MINOR) only emit require(esm) warning under --trace-require-module (Joyee Cheung) #56194 * (SEMVER-MINOR) use synchronous hooks for preparsing in import(cjs) (Joyee Cheung) #55698 * (SEMVER-MINOR) implement module.registerHooks() (Joyee Cheung) #55698 report: * (SEMVER-MINOR) fix typos in report keys and bump the version (Yuan-Ming Hsu) #56068 sqlite: * (SEMVER-MINOR) aggregate constants in a single property (Edigleysson Silva (Edy)) #56213 src,lib: * (SEMVER-MINOR) stabilize permission model (Rafael Gonzaga) #56201 PR-URL: #56310
Notable changes: crypto: * graduate WebCryptoAPI Ed25519 and X25519 algorithms as stable (Filip Skokan) #56142 dgram: * (SEMVER-MINOR) support blocklist in udp (theanarkh) #56087 doc: * stabilize util.styleText (Rafael Gonzaga) #56265 module: * (SEMVER-MINOR) add prefix-only modules to `module.builtinModules` (Jordan Harband) #56185 * (SEMVER-MINOR) only emit require(esm) warning under --trace-require-module (Joyee Cheung) #56194 * (SEMVER-MINOR) use synchronous hooks for preparsing in import(cjs) (Joyee Cheung) #55698 * (SEMVER-MINOR) implement module.registerHooks() (Joyee Cheung) #55698 report: * (SEMVER-MINOR) fix typos in report keys and bump the version (Yuan-Ming Hsu) #56068 sqlite: * (SEMVER-MINOR) aggregate constants in a single property (Edigleysson Silva (Edy)) #56213 src,lib: * (SEMVER-MINOR) stabilize permission model (Rafael Gonzaga) #56201 PR-URL: #56310
This introduces
module.registerHooks()
for registering module loader customization hooks that are run for all modules loaded byrequire()
,import
and functions returned bycreateRequire()
in the same thread, which makes them easier for CJS monkey-patchers to migrate to.This complements the
module.register()
hooks - the new hooks fit better internally and cover all corners in the module graph; whereasmodule.register()
previously could not coverrequire()
while it was in-thread, and still cannot covercreateRequire()
after being moved off-thread.They are also run in the same thread as the modules being loaded and where the hooks are registered, which means they are easier to debug (no more
console.log()
getting lost) and do not have the many deadlock issues haunting themodule.register()
hooks. The new API also takes functions directly so that it's easier for intermediate loader packages to take user options from files that the hooks can't be aware of, like many existing CJS monkey-patchers do.Background
This implements part of the proposal in nodejs/loaders#198 - for the motivations, see #52219 - this fills in the gap where being able to run in-thread and support
require()
is more important for a hook than being able to run asynchronous code (even then, packages like https://www.npmjs.com/package/everysync allows user to sync-ify async code off-thread themselves), especially for a large amount of CJS loader moneky-patchers out there to migrate to an officially supported API easily.The synchronous, in-thread hooks are a lot easier to support universally in the loaders and reduce the number of dark corners where the hooks cannot run or behave in a surprising manner - I guess the amount of "this caveat of asynchronous hooks does not apply to the synchronous hooks" added in the documentation kind of also already proves the point...
For now only
resolve
andload
are implemented to reach parity withmodule.register()
. I left the exports hook to https://github.com/joyeecheung/node/tree/export-hooks which will be a follow-up when we decide the timing at which the post-load hook for CJS should run.The commit
module: use synchronous hooks for preparsing in import(cjs)
isn't strictly necessary and could split into a follow-up instead, but I think it's nice to have it here instead of leaving out that corner uncovered.