module: resolve and instantiate loader pipeline hooks #15445

guybedford · 2017-09-17T10:29:56Z

This PR provides an implementation of a hooking mechanism worked on with some early feedback from @bmeck.

Usage

The es module loading pipeline is hooked via a --loader flag to Node, which specifies a file containing a loader definition.

For example usage:

A loader that loads all files (including .js) as es modules is provided at https://github.com/guybedford/node/blob/resolve-hook-rebased/test/fixtures/es-module-loaders/js-loader.mjs.
A loader that provides named exports for NodeJS builtin modules is provided at https://github.com/guybedford/node/blob/resolve-hook-rebased/test/fixtures/es-module-loaders/builtin-named-exports-loader.mjs.

Motivation

This work is built to follow principles of extensibility - allowing users (framework and platform developers for this feature) to customize their own hooks for their use case needs. The core ecosystem conventions for loading modules in NodeJS will remain the ones that NodeJS sets itself by the ecosystem popularity due to interop requirements.

The goal here though is that through having the ability for users to experiment with loader features will allow the most useful and popular features to potentially be merged back into NodeJS core to extend the ecosystem conventions. So roughly the extensible manifesto thinking.

I believe enabling these use cases is important to avoid possible ecosystem fragmentation through other means.

The examples above show common use cases, and these hooks are also sufficient to implement the package.json "module" property resolution concept using a loader (nodejs/node-eps#60).

Architecture

The hook architecture here is inspired by learnings from the WhatWG loader work, while aiming to cut down the scope of hooking to a bare minimum API surface area, as can be seen from the examples.

The loader itself is given control of module resolution, of the form:

resolve (specifier: string, parentModuleUrl: string) => { url: string, format: string }

Where the "format" returned is "esm" for the ES module loader, "cjs" for the traditional cjs loader, "builtin" to refer to a NodeJS core module, "json" for json and "addon" for a Node binary addon. In future this could be extended for new module formats (ie wasm, binary ast).

This resolver follows exactly what has always been the resolve hook in the module loader specifications, except instead of just returning a string URL, it returns the URL along with the format. Other information could also possibly be returned on this object in future including for example integrity metadata.

When the format returned is "dynamic", then we delegate loading to a dynamicInstantiate hook, hooked in the same way as resolve:

async dynamicInstantiate (url: string) => ({
  exports: string[],
  execute: (exports) => {
    // get and set functions provided for pre-allocated export names
    exports.x.set('asdf');
  }
})

This API is designed to enable the above example use cases, while ensuring that loader internals are kept private, to ensure that what is hooked is only what is intended to be hooked. For example, none of ModuleWrap is exposed.

Feedback very welcome!

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

Affected core subsystem(s)

esmodules

bcoe · 2017-09-17T13:37:34Z

@guybedford what might it look like to extend on the hooks available, such that a modified source-file could be stored in the cache?

I'm interested in how we'll continue providing test-coverage for .mjs files (which I believe circumvent the current require.extensions API -- this seems like it could be a step in the direction of what I'd need to get Istnabul working with .mjs?

guybedford · 2017-09-17T13:54:24Z

@bcoe one option might be to add a separate 'fetch' hook to such a loader object alongside the 'resolve' here which could then add instrumentation. Happy to prototype it if there's interest.

bcoe · 2017-09-17T13:57:59Z

@guybedford I'm very interested. The built-in coverage in V8 doesn't seem to quite be over the finish line yet, so this would be a great transitional step for continuing to support test-coverage with Istanbul.

bcoe · 2017-09-17T21:00:51Z

@guybedford @bmeck any chance we could also allow you to attach a loader in a way similar to module.extensions. Perhaps module.loader? This would make the implementation way easier for tools like Istanbul. Then I could simply check whether module.loader exists, and instrument code accordingly.

guybedford · 2017-09-18T10:43:49Z

@bcoe something like module.loaderHooks = { fetch, resolve } or module.registerHook('resolve', resolve) is possible, but there is the async load timing issue here in that the boot of the application will not run through the hooks in that case. The boot for such a scenario then might look like:

import module from 'module';
import { customResolve, customFetch } from 'custom-loader';

module.registerHook('resolve', customResolve);
module.registerHook('fetch', customFetch);

// now kick off the dynamic import hooking the main application tree load
import('./entry-point.mjs');

This was actually roughly what I had for the original API here, but after discussion with @bmeck we adopted a flag rather here to avoid some of the above issues. I'm still open to either approach.

jasnell · 2017-09-18T14:25:13Z

lib/internal/errors.js

@@ -287,6 +287,8 @@ E('ERR_TRANSFORM_WITH_LENGTH_0',
 E('ERR_UNESCAPED_CHARACTERS',
  (name) => `${name} contains unescaped characters`);
 E('ERR_UNKNOWN_ENCODING', 'Unknown encoding: %s');
+E('ERR_UNKNOWN_FILE_EXTENSION', 'Uknown file extension: %s');


typo: s/Uknown/Unknown

Maybe this could print out the file too?

Sure, instead of just outputting the extension we can output the full path.

jasnell · 2017-09-18T14:25:27Z

lib/internal/errors.js

@@ -287,6 +287,8 @@ E('ERR_TRANSFORM_WITH_LENGTH_0',
 E('ERR_UNESCAPED_CHARACTERS',
  (name) => `${name} contains unescaped characters`);
 E('ERR_UNKNOWN_ENCODING', 'Unknown encoding: %s');
+E('ERR_UNKNOWN_FILE_EXTENSION', 'Uknown file extension: %s');
+E('ERR_UNKNOWN_LOADER', 'Uknown module loader: %s');


here too :-)

The message here seems a bit vague - what does this mean?

jasnell · 2017-09-18T14:28:26Z

src/node.cc

@@ -3950,6 +3955,8 @@ static void CheckIfAllowedInEnv(const char* exe, bool is_env,
    "--no-warnings",
    "--napi-modules",
    "--expose-http2",
+    "--experimental-modules",
+    "--loader",


Perhaps --module-loader would be more descriptive?

@jasnell what might it be confused by?

jasnell · 2017-09-18T14:29:15Z

src/node_config.cc

    READONLY_BOOLEAN_PROPERTY("experimentalModules");
+    if (!config_userland_loader.empty()) {


thank you for using node_config to expose this :-)

That is all @bmeck!

jasnell · 2017-09-18T14:30:31Z

Awesome to see this @guybedford ... left a few initial review comments but will need to play with this a bit more before signing off. Thank you for working on this

bmeck · 2017-09-18T14:46:49Z

I'd like to leave things as declarative instead of API based if possible for now. require.extensions and people monkey patching module.* have cause a lot of lock-in that we cannot undo in CJS in part because of how hard it is to tell when something is being used, and in part because of the power of making them JS object properties.

As for a fetch transform, I have several ideas on how to accomplish similar ideas without introducing that transform:

make a file on disk w/ mutated source and resolve to that.
we introduce blob backed URL.createObjectURL and use that during resolution.

These can be worked through in the browser if done carefully; I am not convinced we should introduce fetch hooks in Node without creating a story for network backed modules and security of them. I also would like some involvement from browser people who are creating multiple different execution workers like ServiceWorker in order to do these tasks.

For code coverage I know V8 now has that instrumented in the codegen, so perhaps using that and adding missing features would be easier over time than trying to have hooks. Especially since we don't have a composition mechanism for these hooks currently.

This PR looks good even though I think we have more than enough hook area already and should not expand the scope of the PR. Overall, I think minimal hooks is the best approach in the beginning as I still think these hooks will change as we use them / won't look exactly the same by the time they are released.

feature	opinion	notes
Resolve hook	+1	need composition eventually
Evaluation hook	+1	would like more research over time into properly managing live bindings for core (e.g. https://github.com/bmeck/node/tree/named-export-core using a Proxy)
Fetch hook	-1	seems like alternates
JS based API	-1	has been a source of pain in past, no clear use case need for it vs declarative

domenic · 2017-09-18T15:26:21Z

I'm not sure when this is targeted to land, but this seems very early. I would hope this doesn't end up in Node core at least until the ecosystem has had some time to settle down and explore alternatives and prove that this particular API shape is the best, and that the use cases it addresses are actually important enough to add such a powerful, fragmentation-encouraging primitive which cannot be taken back.

General API comment: please never return or accept URL objects; see https://url.spec.whatwg.org/#url-apis-elsewhere .

On the actual capabilities here, I think the resolve hook is straightforward enough (although may end up mismatching browsers if they need something more declarative, as is currently the thinking). The loader object is much scarier and is nowhere near what we would currently contemplate exposing in browsers (as essentially we are trying to stick with Source Text Module Records, so any hook I can imagine would be just a fetch hook, i.e. service workers).

Fishrock123

left some comments :)

Fishrock123 · 2017-09-18T15:27:48Z

lib/internal/errors.js

@@ -287,6 +287,8 @@ E('ERR_TRANSFORM_WITH_LENGTH_0',
 E('ERR_UNESCAPED_CHARACTERS',
  (name) => `${name} contains unescaped characters`);
 E('ERR_UNKNOWN_ENCODING', 'Unknown encoding: %s');
+E('ERR_UNKNOWN_FILE_EXTENSION', 'Uknown file extension: %s');
+E('ERR_UNKNOWN_LOADER', 'Uknown module loader: %s');


The message here seems a bit vague - what does this mean?

Fishrock123 · 2017-09-18T15:28:16Z

lib/internal/errors.js

@@ -287,6 +287,8 @@ E('ERR_TRANSFORM_WITH_LENGTH_0',
 E('ERR_UNESCAPED_CHARACTERS',
  (name) => `${name} contains unescaped characters`);
 E('ERR_UNKNOWN_ENCODING', 'Unknown encoding: %s');
+E('ERR_UNKNOWN_FILE_EXTENSION', 'Uknown file extension: %s');


Maybe this could print out the file too?

Fishrock123 · 2017-09-18T15:39:11Z

lib/internal/loader/ModuleRequest.js

+
+// Strategy for loading a standard JavaScript module
+loaders.set('esm', async (url) => {
+  const source = `${await asyncReadFile(url)}`;


What happens if the readFile throws?

The loader fails to produce a ModuleWrap

The module job it is placed in produces an error (cached as per spec): https://github.com/nodejs/node/pull/15445/files/db6eb0069cfc94d00b647525a47ddc73dba9ccd8#diff-e6f2798544dae1f6ec863621e745489aR93

All imports (current and in future) of that absolute url fail due to the job failing: https://github.com/nodejs/node/pull/15445/files/db6eb0069cfc94d00b647525a47ddc73dba9ccd8#diff-e6f2798544dae1f6ec863621e745489aR101

Fishrock123 · 2017-09-18T15:41:39Z

lib/internal/loader/ModuleRequest.js

+      return { url, loader: 'esm' };
+    case '.json':
+      return { url, loader: 'cjs' };
+    case '.node':


Aren't .node files native addon C++?

they use the CJS interop

It might be nice to add a binary format now - suggestions welcome for how to load a file as just a Node binary, throwing otherwise.

Fishrock123 · 2017-09-18T15:42:16Z

lib/internal/loader/ModuleRequest.js

+    case '.js':
+      return { url, loader: 'cjs' };
+    default:
+      throw new errors.Error('ERR_UNKNOWN_FILE_EXTENSION', ext);


Shouldn't this default to 'cjs'? (Though I suppose this is not required for importing CJS via ESM...)

This is for future proofing, just like #15365 was talking about.

Fishrock123 · 2017-09-18T15:45:52Z

src/node.cc

@@ -3950,6 +3955,8 @@ static void CheckIfAllowedInEnv(const char* exe, bool is_env,
    "--no-warnings",
    "--napi-modules",
    "--expose-http2",
+    "--experimental-modules",
+    "--loader",


@jasnell what might it be confused by?

guybedford · 2017-09-18T18:19:01Z

Thanks so much for all feedback so far. I've updated the description above to match the latest updates:

I've renamed loader to format in the resolver hook, which is now fully consistent with being interpreted as the module format of the module being loaded, and no longer accepts a loader function. Hopefully this simplifies the resolve hook and makes it easier to see it as something that might be compatible with browsers too.
To support the previous loader option, when the "dynamic" format is returned, module loading is delegated to a separate dynamicInstantiate hook which takes the place of the functional loader option.
The hooks no longer accept or return URL objects, but work with string URLs only.

benjamn · 2017-09-18T20:45:55Z

lib/module.js

@@ -490,6 +490,10 @@ function tryModuleLoad(module, filename) {
  }
 }

+Module.import = function(request, parent) {
+  return ESMLoader.import(request, parent);
+};


Why not make this Module.prototype.import so you can get rid of the parent parameter (and use this instead)?

there eventually could be multiple loaders / I would like to see this fn removed when import() lands

Module.import doesn't appear to be used anywhere else in this PR or usage examples, is it dead code?

used as stop gap til import() in V8@6.2 lands

used as stop gap til import() in V8@6.2 lands

Is there a rough idea of when that'll be?
Would Module.import be intended as public consumable API?

Sometime in Node@9
no, i'd remove it once 6.2 lands.

We should guard this by checking for experimentalModules

iarna · 2017-09-19T00:26:44Z

Ok, so @bcoe asked me to chime in:

My primary use case for a custom loader is mocks. I currently maintain require-inject which fiddles with the cache to allow tests to replace modules with their own code. How it does this is not something I'd want to see replicated as it's very much a hack, but the functionality is something I don't want to lose with ESM.

It looks like a loader API + Module.import() or a v8 import() would meet my needs.

An interpreter flag might be workable (because these are tests) but would be awkward, and some version of import() would still be needed. (Currently I can run my tests with node ./path/to/test.js, having to run it as node --with-loader=./path/to/loader.js ./path/to/test.mjs would basically make that kind of ad hoc command line use too awkward to use.)

jdalton · 2017-09-19T00:44:31Z

@iarna

having to run it as node --with-loader=./path/to/loader.js ./path/to/test.mjs would basically make that kind of ad hoc command line use too awkward to use.)

I'd assume --loader package-name works too, which is similar to how I'm seeing folks use @std/esm (using node -r @std/esm ./path/to/test.js). Would that smooth things over?

TimothyGu · 2017-09-19T00:42:38Z

lib/internal/loader/Loader.js

+      parentURLOrString = new URL(parentURLOrString);
+    } else if (parentURLOrString instanceof URL === false) {
+      throw new errors.TypeError('ERR_INVALID_ARG_TYPE',
+                                 'parentURLOrString', 'URL');


This doesn't really address Domenic's concern of a mutable URL object being passed around. Instead you can do if (typeof parentURLOrString !== 'string' && !(parentURLOrString instanceof URL)) throw ...; const url = new URL(parentURLOrString);.

Or just do ToString on any incoming argument like every ES and web API do.

I think the Node code precedent here is to use template strings to coerce:

`${url}`

This resolve isn't exposed publicly, it's the this.resolver call just below that represents the hookable API. Will add a guard to that.

bcoe · 2017-09-19T01:01:17Z

@jdalton @iarna

I'd assume --loader package-name works too, which is similar to how I'm seeing folks use @std/esm (using node -r @std/esm ./path/to/test.js). Would that smooth things over?

This becomes a bit awkward for writing testing tools like nyc. Users will often spawn a subprocess in a cwd that potentially has its own node_modules folder. So, unless you were going to require that someone install this category of tool globally (which feels gross), you end up doing a lot of bookkeeping to ensure that spawned subprocesses have the appropriate flags.

This is a similar concern I have with trying to facilitate test-coverage functionality using V8's built-in coverage. Which is even more of a Rube Goldberg machine, as you need to spawn a process with --inspect, track which port has been used, etc.

tldr; it's a lot easier if there's a programatic way hook into this functionality in a vanilla Node process .

jdalton · 2017-09-19T01:08:52Z

Users will often spawn a subprocess in a cwd that potentially has its own node_modules folder. So, unless you were going to require that someone install this category of tool globally (which feels gross)

I don't have a ton of experience with all that. I know that using tools like AVA, which forks processes, I was able to enable @std/esm by tying into their require option. Which is to say if AVA can track requires (and I'd guess flags too) couldn't nyc and others too?

bcoe · 2017-09-19T01:17:29Z

@jdalton

I don't have a ton of experience with all that. I know that using tools like AVA, which forks processes, I was able to enable @std/esm by tying into their require option (which is serialized as a JSON string process.argv arg). Which is to say if AVA can do it couldn't nyc and others too?

nyc already does a lot of this, passing around state so that subprocesses can when spawned have their require.extensions patched (so that including a source-file results in it being instrumented by Istanbul).

My concern is that this flow is already very complex and bug prone. As it stands right now the new module system makes writing a development tool like this even more difficult.

EDIT: tldr; it feels like there's an opportunity to make the API for building features like this less painful, since the loader implementation is a greenfield right now. So, I don't see it as a question of whether it's possible, it's a question of can we reduce the burden placed on folks developing tools for Node.js.

bmeck · 2017-09-19T01:38:37Z

@bcoe

This becomes a bit awkward for writing testing tools like nyc. Users will often spawn a subprocess in a cwd that potentially has its own node_modules folder. So, unless you were going to require that someone install this category of tool globally (which feels gross), you end up doing a lot of bookkeeping to ensure that spawned subprocesses have the appropriate flags.

You can use the NODE_OPTIONS environment variable to intercept the subprocess and attach a --loader argument. Is that sufficient?

This is a similar concern I have with trying to facilitate test-coverage functionality using V8's built-in coverage. Which is even more of a Rube Goldberg machine, as you need to spawn a process with --inspect, track which port has been used, etc.

This seems like we can improve that workflow rather than making a more complex hook system.

My concern is that this flow is already very complex and bug prone. As it stands right now the new module system makes writing a development tool like this even more difficult.

This is exactly why I want to work on a minimal API and iterate rather than trying to recreate the old workflows. If we focus on specific aspects instead of trying to solve all the use cases at once I think we will not suffer the previous problems with standardizing loaders.

bcoe · 2017-09-19T01:53:51Z

@bmeck

That's a really thoughtful point of view; mainly I just want to make sure we have examples of how we'd facilitate use-cases like dependency injection, coverage, and transpilation, before ES modules aren't behind a flag... I think if we can nail these three categories of common developer tools, we'd know the API changes are looking really healthy.

bmeck · 2017-09-19T14:04:31Z

@bcoe Dependency injection won't work like in other languages where it can change over time due to idempotency of ESM for any given specifier. The spec does not allow things like:

await import('thingFactory') != await import('thingFactory');

You can pass in data via query params as needed and that will give you a clean local module map that can change what thingFactory is. Can't re-use the same module though.

I'm not going to block unflagging on transpilation as they can be done ahead of time. However, various workflows like needing a way to get coverage is certainly blocking. If that is done ahead of time and works I would not see it as a blocker.

bmeck · 2017-09-19T14:09:26Z

lib/module.js

@@ -490,6 +490,10 @@ function tryModuleLoad(module, filename) {
  }
 }

+Module.import = function(request, parent) {
+  return ESMLoader.import(request, parent);
+};


We should guard this by checking for experimentalModules

Trott · 2017-10-12T17:43:16Z

~~Unfortunately, it appears this landed without a CI run 😱~~

~~...and it fails on Raspberry Pi devices:~~

The test added here has started failing on Raspberry Pi devices.

https://ci.nodejs.org/job/node-test-binary-arm/10851/RUN_SUBSET=4,label=pi1-raspbian-wheezy/console

not ok 249 es-module/test-esm-addon
  ---
  duration_ms: 2.65
  severity: fail
  stack: |-
    (node:29865) ExperimentalWarning: The ESM module loader is experimental.
    Error: module ../addons/hello-world/build/Release/binding.node not found
        at module.exports (internal/loader/search.js:14:12)
        at exports.resolve (internal/loader/ModuleRequest.js:91:13)
        at Loader.resolve (internal/loader/Loader.js:50:40)
        at Loader.getModuleJob (internal/loader/Loader.js:78:40)
        at ModuleWrap.module.link (internal/loader/ModuleJob.js:27:25)
        at linked (internal/loader/ModuleJob.js:25:19)
        at <anonymous>

Trott · 2017-10-12T17:44:34Z

Oh, wait, I'm totally wrong, it had CI, even passing CI. (I searched for CI: rather than just CI because I'm a chump.) Sorry about that.

OK, so wonder what changed to cause the rash of failures all of a sudden...

bcoe · 2017-10-12T17:51:38Z

@bmeck where would be a good place to start (or continue) a discussion about createObjectURL; I think this would be the most promising short term solution for keeping nyc/Istanbul working as we start to move some of the Node core modules over to .mjs.

Trott · 2017-10-12T18:12:40Z

Talking with @refack in IRC. The issue is indeed with this change but CI didn't catch it because of a bug in Makefile that caused the test to be omitted on CI.

Most relevant bit:

Yeah. This can't work on the ARM

node/test/es-module/test-esm-addon.mjs

Line 5 in 537402f

import binding from '../addons/hello-world/build/Release/binding.node';

And is also build-order-sensitive on all other platforms

Trott · 2017-10-12T18:43:21Z

Ref: #16155 (comment)

refack · 2017-10-12T19:26:28Z

Yeah. This can't work on the ARM
And is also build-order-sensitive on all other platforms

For reference the arm-fanned job runs the addons suite on a different machine than any of the JS tests, so there is no compilation of the hello-world addon before test/es-module/test-esm-addon.mjs is run. (and also on the other platforms it's only by chance that it is done before this test is run).
tl;dr is should be moved to test/addons/hello-world/

Trott · 2017-10-13T05:06:48Z

Proposed fix in #16174

This enables a --loader flag for Node, which can provide custom "resolve" and "dynamicInstantiate" methods for custom ES module loading. In the process, module providers have been converted from classes into functions and the module APIs have been made to pass URL strings over objects. PR-URL: #15445 Reviewed-By: Bradley Farias <bradley.meck@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com> Reviewed-By: Michaël Zasso <targos@protonmail.com> Reviewed-By: Timothy Gu <timothygu99@gmail.com>

Add `"dynamic"` to the list of supported `format`s returned by a custom resolve hook. Add a table describing the meaning of each `format`. Refs: nodejs#15445

Notable Changes: * crypto: - expose ECDH class #8188 * http2: - http2 is now exposed by defualt without the need for a flag #15685 - a new environment varible NODE\_NO\_HTTP2 has been added to allow userland http2 to be required #15685 - support has been added for generic `Duplex` streams #16269 * module: - resolve and instantiate loader pipeline hooks have been added to the ESM lifecycle #15445 * zlib: - CVE-2017-14919 - In zlib v1.2.9, a change was made that causes an error to be raised when a raw deflate stream is initialized with windowBits set to 8. On some versions this crashes Node and you cannot recover from it, while on some versions it throws an exception. Node.js will now gracefully set windowBits to 9 replicating the legacy behavior to avoid a DOS vector. https://github.com/nodejs-private/node-private/pull/95 PR-URL: https://github.com/nodejs-private/node-private/pull/98

Notable Changes: * crypto: - expose ECDH class nodejs/node#8188 * http2: - http2 is now exposed by defualt without the need for a flag nodejs/node#15685 - a new environment varible NODE\_NO\_HTTP2 has been added to allow userland http2 to be required nodejs/node#15685 - support has been added for generic `Duplex` streams nodejs/node#16269 * module: - resolve and instantiate loader pipeline hooks have been added to the ESM lifecycle nodejs/node#15445 * zlib: - CVE-2017-14919 - In zlib v1.2.9, a change was made that causes an error to be raised when a raw deflate stream is initialized with windowBits set to 8. On some versions this crashes Node and you cannot recover from it, while on some versions it throws an exception. Node.js will now gracefully set windowBits to 9 replicating the legacy behavior to avoid a DOS vector. nodejs-private/node-private#95 PR-URL: nodejs-private/node-private#98

Add `"dynamic"` to the list of supported `format`s returned by a custom resolve hook. Add a table describing the meaning of each `format`. PR-URL: #16375 Refs: #15445 Reviewed-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com> Reviewed-By: Yuta Hiroto <hello@about-hiroppy.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Michaël Zasso <targos@protonmail.com>

Notable Changes: * crypto: - expose ECDH class nodejs/node#8188 * http2: - http2 is now exposed by defualt without the need for a flag nodejs/node#15685 - a new environment varible NODE\_NO\_HTTP2 has been added to allow userland http2 to be required nodejs/node#15685 - support has been added for generic `Duplex` streams nodejs/node#16269 * module: - resolve and instantiate loader pipeline hooks have been added to the ESM lifecycle nodejs/node#15445 * zlib: - CVE-2017-14919 - In zlib v1.2.9, a change was made that causes an error to be raised when a raw deflate stream is initialized with windowBits set to 8. On some versions this crashes Node and you cannot recover from it, while on some versions it throws an exception. Node.js will now gracefully set windowBits to 9 replicating the legacy behavior to avoid a DOS vector. nodejs-private/node-private#95 PR-URL: nodejs-private/node-private#98

Add `"dynamic"` to the list of supported `format`s returned by a custom resolve hook. Add a table describing the meaning of each `format`. PR-URL: #16375 Refs: #15445 Reviewed-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com> Reviewed-By: Yuta Hiroto <hello@about-hiroppy.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Michaël Zasso <targos@protonmail.com>

nodejs-github-bot added the lib / src Issues and PRs related to general changes in the lib or src directory. label Sep 17, 2017

guybedford changed the title ~~Loader pipeline resolver and loader hooking~~ Module pipeline resolver and loader hooking Sep 17, 2017

bcoe mentioned this pull request Sep 17, 2017

build: run es-module tests in ci environment #15276

Closed

3 tasks

mscdex added the esm Issues and PRs related to the ECMAScript Modules implementation. label Sep 17, 2017

jasnell reviewed Sep 18, 2017

View reviewed changes

Fishrock123 reviewed Sep 18, 2017

View reviewed changes

benjamn reviewed Sep 18, 2017

View reviewed changes

bmeck mentioned this pull request Sep 18, 2017

Consider additive systems for importing ES Modules ayojs/ayo#56

Open

TimothyGu reviewed Sep 19, 2017

View reviewed changes

bmeck suggested changes Sep 19, 2017

View reviewed changes

Trott mentioned this pull request Oct 12, 2017

test-esm-addon.mjs fails when building in debug mode #16155

Closed

demurgos mentioned this pull request Oct 13, 2017

.mjs extension trade-offs revision nodejs/node-eps#57

Closed

bcoe mentioned this pull request Oct 13, 2017

Implement createObjectURL/Blob from File API #16167

Closed

5 tasks

jdalton mentioned this pull request Oct 19, 2017

Add new pragma to transform-es2015-modules babel/babel#6449

Closed

azz mentioned this pull request Oct 22, 2017

doc: document resolve hook formats #16375

Closed

2 tasks

azz added a commit to azz/node that referenced this pull request Oct 22, 2017

doc: document resolve hook formats

f4121de

Add `"dynamic"` to the list of supported `format`s returned by a custom resolve hook. Add a table describing the meaning of each `format`. Refs: nodejs#15445

teppeis mentioned this pull request Oct 24, 2017

doc: remove loader hooks from unsupported features #16465

Closed

4 tasks

guybedford mentioned this pull request Nov 12, 2017

loader: refactor loader #16874

Closed

3 tasks

aduh95 mentioned this pull request Apr 18, 2022

doc: fix version history for Loaders API #42778

Merged

		READONLY_BOOLEAN_PROPERTY("experimentalModules");
		if (!config_userland_loader.empty()) {

module: resolve and instantiate loader pipeline hooks #15445

module: resolve and instantiate loader pipeline hooks #15445

Conversation

guybedford commented Sep 17, 2017 • edited Loading

Usage

Motivation

Architecture

Checklist

Affected core subsystem(s)

bcoe commented Sep 17, 2017

guybedford commented Sep 17, 2017

bcoe commented Sep 17, 2017

bcoe commented Sep 17, 2017

guybedford commented Sep 18, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasnell commented Sep 18, 2017

bmeck commented Sep 18, 2017

domenic commented Sep 18, 2017

Fishrock123 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guybedford commented Sep 18, 2017

Choose a reason for hiding this comment

bmeck Sep 18, 2017 • edited Loading

Choose a reason for hiding this comment

jdalton Sep 18, 2017 • edited Loading

Choose a reason for hiding this comment

bmeck Sep 18, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iarna commented Sep 19, 2017 • edited Loading

jdalton commented Sep 19, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bcoe commented Sep 19, 2017 • edited Loading

jdalton commented Sep 19, 2017 • edited Loading

bcoe commented Sep 19, 2017 • edited Loading

bmeck commented Sep 19, 2017

bcoe commented Sep 19, 2017

bmeck commented Sep 19, 2017

Choose a reason for hiding this comment

Trott commented Oct 12, 2017 • edited Loading

Trott commented Oct 12, 2017

bcoe commented Oct 12, 2017

Trott commented Oct 12, 2017 • edited Loading

Trott commented Oct 12, 2017

refack commented Oct 12, 2017 • edited Loading

Trott commented Oct 13, 2017

guybedford commented Sep 17, 2017 •

edited

Loading

guybedford commented Sep 18, 2017 •

edited

Loading

bmeck Sep 18, 2017 •

edited

Loading

jdalton Sep 18, 2017 •

edited

Loading

bmeck Sep 18, 2017 •

edited

Loading

iarna commented Sep 19, 2017 •

edited

Loading

jdalton commented Sep 19, 2017 •

edited

Loading

bcoe commented Sep 19, 2017 •

edited

Loading

jdalton commented Sep 19, 2017 •

edited

Loading

bcoe commented Sep 19, 2017 •

edited

Loading

Trott commented Oct 12, 2017 •

edited

Loading

Trott commented Oct 12, 2017 •

edited

Loading

refack commented Oct 12, 2017 •

edited

Loading