esm: irp type implementation

Refs: https://github.com/GeoffreyBooth/node-import-file-specifier-resolution-proposal
Refs: nodejs/modules#180
Refs: #6
Refs: #12
Refs: #28
Co-authored-by: Myles Borins <MylesBorins@google.com>
Co-authored-by: John-David Dalton <john.david.dalton@gmail.com>

Author: 3 people

.eslintrc.js

@@ -38,6 +38,8 @@ module.exports = {
     {
       files: [
         'doc/api/esm.md',
+        'test/es-module/test-esm-type-flag.js',
+        'test/es-module/test-esm-type-flag-alias.js',
         '*.mjs',
         'test/es-module/test-esm-example-loader.js',
       ],

doc/api/cli.md

@@ -511,6 +511,18 @@ added: v2.4.0
 
 Track heap object allocations for heap snapshots.
 
+### `-m`, `--type=type`
+
+When using `--experimental-modules`, this informs the module resolution type
+to interpret the top-level entry into Node.js.
+
+Works with stdin, `--eval`, `--print` as well as standard execution.
+
+Valid values are `"commonjs"` and `"module"`, where the default is to infer
+from the file extension and package type boundary.
+
+`-m` is an alias for `--type=module`.
+
 ### `--use-bundled-ca`, `--use-openssl-ca`
 <!-- YAML
 added: v6.11.0
@@ -904,7 +916,7 @@ greater than `4` (its current default value). For more information, see the
 [debugger]: debugger.html
 [debugging security implications]: https://nodejs.org/en/docs/guides/debugging-getting-started/#security-implications
 [emit_warning]: process.html#process_process_emitwarning_warning_type_code_ctor
-[experimental ECMAScript Module]: esm.html#esm_loader_hooks
+[experimental ECMAScript Module]: esm.html#esm_experimental_loader_hooks
 [libuv threadpool documentation]: http://docs.libuv.org/en/latest/threadpool.html
 [remote code execution]: https://www.owasp.org/index.php/Code_Injection
 [secureProtocol]: tls.html#tls_tls_createsecurecontext_options

doc/api/errors.md

@@ -1267,6 +1267,11 @@ An invalid or unexpected value was passed in an options object.
 
 An invalid or unknown file encoding was passed.
 
+<a id="ERR_INVALID_PACKAGE_CONFIG"></a>
+### ERR_INVALID_PACKAGE_CONFIG
+
+An invalid `package.json` file was found which failed parsing.
+
 <a id="ERR_INVALID_PERFORMANCE_MARK"></a>
 ### ERR_INVALID_PERFORMANCE_MARK
 
@@ -1449,26 +1454,19 @@ a `dynamicInstantiate` hook.
 A `MessagePort` was found in the object passed to a `postMessage()` call,
 but not provided in the `transferList` for that call.
 
-<a id="ERR_MISSING_MODULE"></a>
-### ERR_MISSING_MODULE
-
-> Stability: 1 - Experimental
-
-An [ES6 module][] could not be resolved.
-
 <a id="ERR_MISSING_PLATFORM_FOR_WORKER"></a>
 ### ERR_MISSING_PLATFORM_FOR_WORKER
 
 The V8 platform used by this instance of Node.js does not support creating
 Workers. This is caused by lack of embedder support for Workers. In particular,
 this error will not occur with standard builds of Node.js.
 
-<a id="ERR_MODULE_RESOLUTION_LEGACY"></a>
-### ERR_MODULE_RESOLUTION_LEGACY
+<a id="ERR_MODULE_NOT_FOUND"></a>
+### ERR_MODULE_NOT_FOUND
 
 > Stability: 1 - Experimental
 
-A failure occurred resolving imports in an [ES6 module][].
+An [ESM module][] could not be resolved.
 
 <a id="ERR_MULTIPLE_CALLBACK"></a>
 ### ERR_MULTIPLE_CALLBACK
@@ -2220,6 +2218,32 @@ A non-specific HTTP/2 error has occurred.
 Used in the `repl` in case the old history file is used and an error occurred
 while trying to read and parse it.
 
+<a id="ERR_INVALID_REPL_TYPE"></a>
+### ERR_INVALID_REPL_TYPE
+
+> Stability: 1 - Experimental
+
+The `--type=...` flag is not compatible with the Node.js REPL.
+
+<a id="ERR_TYPE_MISMATCH"></a>
+### ERR_TYPE_MISMATCH
+
+> Stability: 1 - Experimental
+
+The `--type=commonjs` flag was used to attempt to execute an `.mjs` file or
+a `.js` file where the nearest parent `package.json` contains
+`"type": "module"`; or
+the `--type=module` flag was used to attempt to execute a `.cjs` file or
+a `.js` file where the nearest parent `package.json` either lacks a `"type"`
+field or contains `"type": "commonjs"`.
+
+<a id="ERR_INVALID_TYPE_FLAG"></a>
+### ERR_INVALID_TYPE_FLAG
+
+> Stability: 1 - Experimental
+
+An invalid `--type=...` flag value was provided.
+
 <a id="ERR_MISSING_DYNAMIC_INSTANTIATE_HOOK"></a>
 #### ERR_MISSING_DYNAMIC_INSTANTIATE_HOOK
 
@@ -2250,7 +2274,6 @@ size.
 This `Error` is thrown when a read is attempted on a TTY `WriteStream`,
 such as `process.stdout.on('data')`.
 
-
 [`'uncaughtException'`]: process.html#process_event_uncaughtexception
 [`--force-fips`]: cli.html#cli_force_fips
 [`Class: assert.AssertionError`]: assert.html#assert_class_assert_assertionerror

doc/api/esm.md

@@ -8,10 +8,11 @@
 <!--name=esm-->
 
 Node.js contains support for ES Modules based upon the
-[Node.js EP for ES Modules][].
+[Node.js EP for ES Modules][] and the [ESM Minimal Kernel][].
 
-Not all features of the EP are complete and will be landing as both VM support
-and implementation is ready. Error messages are still being polished.
+The minimal feature set is designed to be compatible with all potential
+future implementations. Expect major changes in the implementation including
+interoperability support, specifier resolution, and default behavior.
 
 ## Enabling
 
@@ -54,6 +55,10 @@ property:
 
 ## Notable differences between `import` and `require`
 
+### Mandatory file extensions
+
+You must provide a file extension when using the `import` keyword.
+
 ### No NODE_PATH
 
 `NODE_PATH` is not part of resolving `import` specifiers. Please use symlinks
@@ -78,31 +83,32 @@ Modules will be loaded multiple times if the `import` specifier used to resolve
 them have a different query or fragment.
 
 ```js
-import './foo?query=1'; // loads ./foo with query of "?query=1"
-import './foo?query=2'; // loads ./foo with query of "?query=2"
+import './foo.mjs?query=1'; // loads ./foo.mjs with query of "?query=1"
+import './foo.mjs?query=2'; // loads ./foo.mjs with query of "?query=2"
 ```
 
 For now, only modules using the `file:` protocol can be loaded.
 
-## Interop with existing modules
+## CommonJS, JSON, and Native Modules
 
-All CommonJS, JSON, and C++ modules can be used with `import`.
+CommonJS, JSON, and Native modules can be used with [`module.createRequireFromPath()`][].
 
-Modules loaded this way will only be loaded once, even if their query
-or fragment string differs between `import` statements.
+```js
+// cjs.js
+module.exports = 'cjs';
 
-When loaded via `import` these modules will provide a single `default` export
-representing the value of `module.exports` at the time they finished evaluating.
+// esm.mjs
+import { createRequireFromPath as createRequire } from 'module';
+import { fileURLToPath as fromPath } from 'url';
 
-```js
-// foo.js
-module.exports = { one: 1 };
+const require = createRequire(fromPath(import.meta.url));
 
-// bar.mjs
-import foo from './foo.js';
-foo.one === 1; // true
+const cjs = require('./cjs');
+cjs === 'cjs'; // true
 ```
 
+## Builtin modules
+
 Builtin modules will provide named exports of their public API, as well as a
 default export which can be used for, among other things, modifying the named
 exports. Named exports of builtin modules are updated when the corresponding
@@ -132,7 +138,160 @@ fs.readFileSync = () => Buffer.from('Hello, ESM');
 fs.readFileSync === readFileSync;
 ```
 
-## Loader hooks
+## Resolution Algorithm
+
+### Features
+
+The resolver has the following properties:
+
+* FileURL-based resolution as is used by ES modules
+* Support for builtin module loading
+* Relative and absolute URL resolution
+* No default extensions
+* No folder mains
+* Bare specifier package resolution lookup through node_modules
+
+### Resolver Algorithm
+
+The algorithm to load an ES module specifier is given through the
+**ESM_RESOLVE** method below. It returns the resolved URL for a
+module specifier relative to a parentURL, in addition to the unique module
+format for that resolved URL given by the **ESM_FORMAT** routine.
+
+The _"module"_ format is returned for an ECMAScript Module, while the
+_"commonjs"_ format is used to indicate loading through the legacy
+CommonJS loader. Additional formats such as _"wasm"_ or _"addon"_ can be
+extended in future updates.
+
+In the following algorithms, all subroutine errors are propogated as errors
+of these top-level routines.
+
+_isMain_ is **true** when resolving the Node.js application entry point.
+
+If the top-level `--type` is _"commonjs"_, then the ESM resolver is skipped
+entirely for the CommonJS loader.
+
+If the top-level `--type` is _"module"_, then the ESM resolver is used
+as described here, with the conditional `--type` check in **ESM_FORMAT**.
+
+**ESM_RESOLVE(_specifier_, _parentURL_, _isMain_)**
+> 1. Let _resolvedURL_ be **undefined**.
+> 1. If _specifier_ is a valid URL, then
+>    1. Set _resolvedURL_ to the result of parsing and reserializing
+>       _specifier_ as a URL.
+> 1. Otherwise, if _specifier_ starts with _"/"_, then
+>    1. Throw an _Invalid Specifier_ error.
+> 1. Otherwise, if _specifier_ starts with _"./"_ or _"../"_, then
+>    1. Set _resolvedURL_ to the URL resolution of _specifier_ relative to
+>       _parentURL_.
+> 1. Otherwise,
+>    1. Note: _specifier_ is now a bare specifier.
+>    1. Set _resolvedURL_ the result of
+>       **PACKAGE_RESOLVE**(_specifier_, _parentURL_).
+> 1. If the file at _resolvedURL_ does not exist, then
+>    1. Throw a _Module Not Found_ error.
+> 1. Set _resolvedURL_ to the real path of _resolvedURL_.
+> 1. Let _format_ be the result of **ESM_FORMAT**(_resolvedURL_, _isMain_).
+> 1. Load _resolvedURL_ as module format, _format_.
+
+PACKAGE_RESOLVE(_packageSpecifier_, _parentURL_)
+> 1. Let _packageName_ be *undefined*.
+> 1. Let _packageSubpath_ be *undefined*.
+> 1. If _packageSpecifier_ is an empty string, then
+>    1. Throw an _Invalid Specifier_ error.
+> 1. If _packageSpecifier_ does not start with _"@"_, then
+>    1. Set _packageName_ to the substring of _packageSpecifier_ until the
+>       first _"/"_ separator or the end of the string.
+> 1. Otherwise,
+>    1. If _packageSpecifier_ does not contain a _"/"_ separator, then
+>       1. Throw an _Invalid Specifier_ error.
+>    1. Set _packageName_ to the substring of _packageSpecifier_
+>       until the second _"/"_ separator or the end of the string.
+> 1. Let _packageSubpath_ be the substring of _packageSpecifier_ from the
+>    position at the length of _packageName_ plus one, if any.
+> 1. Assert: _packageName_ is a valid package name or scoped package name.
+> 1. Assert: _packageSubpath_ is either empty, or a path without a leading
+>    separator.
+> 1. If _packageSubpath_ contains any _"."_ or _".."_ segments or percent
+>    encoded strings for _"/"_ or _"\"_ then,
+>    1. Throw an _Invalid Specifier_ error.
+> 1. If _packageSubpath_ is empty and _packageName_ is a Node.js builtin
+>    module, then
+>    1. Return the string _"node:"_ concatenated with _packageSpecifier_.
+> 1. While _parentURL_ is not the file system root,
+>    1. Set _parentURL_ to the parent folder URL of _parentURL_.
+>    1. Let _packageURL_ be the URL resolution of the string concatenation of
+>       _parentURL_, _"/node_modules/"_ and _packageSpecifier_.
+>    1. If the folder at _packageURL_ does not exist, then
+>       1. Set _parentURL_ to the parent URL path of _parentURL_.
+>       1. Continue the next loop iteration.
+>    1. Let _pjson_ be the result of **READ_PACKAGE_JSON**(_packageURL_).
+>    1. If _packageSubpath_ is empty, then
+>       1. Return the result of **PACKAGE_MAIN_RESOLVE**(_packageURL_,
+>          _pjson_).
+>    1. Otherwise,
+>       1. Return the URL resolution of _packageSubpath_ in _packageURL_.
+> 1. Throw a _Module Not Found_ error.
+
+PACKAGE_MAIN_RESOLVE(_packageURL_, _pjson_)
+> 1. If _pjson_ is **null**, then
+>    1. Throw a _Module Not Found_ error.
+> 1. If _pjson.main_ is a String, then
+>    1. Let _resolvedMain_ be the concatenation of _packageURL_, "/", and
+>       _pjson.main_.
+>    1. If the file at _resolvedMain_ exists, then
+>       1. Return _resolvedMain_.
+> 1. If _pjson.type_ is equal to _"module"_, then
+>    1. Throw a _Module Not Found_ error.
+> 1. Let _legacyMainURL_ be the result applying the legacy
+>    **LOAD_AS_DIRECTORY** CommonJS resolver to _packageURL_, throwing a
+>    _Module Not Found_ error for no resolution.
+> 1. If _legacyMainURL_ does not end in _".js"_ then,
+>    1. Throw an _Unsupported File Extension_ error.
+> 1. Return _legacyMainURL_.
+
+**ESM_FORMAT(_url_, _isMain_)**
+> 1. Assert: _url_ corresponds to an existing file.
+> 1. If _isMain_ is **true** and the `--type` flag is _"module"_, then
+>    1. If _url_ ends with _".cjs"_, then
+>       1. Throw a _Type Mismatch_ error.
+>    1. Return _"module"_.
+> 1. Let _pjson_ be the result of **READ_PACKAGE_BOUNDARY**(_url_).
+> 1. If _pjson_ is **null** and _isMain_ is **true**, then
+>    1. If _url_ ends in _".mjs"_, then
+>       1. Return _"module"_.
+>    1. Return _"commonjs"_.
+> 1. If _pjson.type_ exists and is _"module"_, then
+>    1. If _url_ ends in _".cjs"_, then
+>       1. Return _"commonjs"_.
+>    1. Return _"module"_.
+> 1. Otherwise,
+>    1. If _url_ ends in _".mjs"_, then
+>       1. Return _"module"_.
+>    1. If _url_ does not end in _".js"_, then
+>       1. Throw an _Unsupported File Extension_ error.
+>    1. Return _"commonjs"_.
+
+READ_PACKAGE_BOUNDARY(_url_)
+> 1. Let _boundaryURL_ be _url_.
+> 1. While _boundaryURL_ is not the file system root,
+>    1. Let _pjson_ be the result of **READ_PACKAGE_JSON**(_boundaryURL_).
+>    1. If _pjson_ is not **null**, then
+>       1. Return _pjson_.
+>    1. Set _boundaryURL_ to the parent URL of _boundaryURL_.
+> 1. Return **null**.
+
+READ_PACKAGE_JSON(_packageURL_)
+> 1. Let _pjsonURL_ be the resolution of _"package.json"_ within _packageURL_.
+> 1. If the file at _pjsonURL_ does not exist, then
+>    1. Return **null**.
+> 1. If the file at _packageURL_ does not parse as valid JSON, then
+>    1. Throw an _Invalid Package Configuration_ error.
+> 1. Return the parsed JSON source of the file at _pjsonURL_.
+
+## Experimental Loader hooks
+
+**Note: This API is currently being redesigned and will still change.**.
 
 <!-- type=misc -->
 
@@ -173,11 +332,9 @@ module. This can be one of the following:
 
 | `format` | Description |
 | --- | --- |
-| `'esm'` | Load a standard JavaScript module |
-| `'cjs'` | Load a node-style CommonJS module |
-| `'builtin'` | Load a node builtin CommonJS module |
-| `'json'` | Load a JSON file |
-| `'addon'` | Load a [C++ Addon][addons] |
+| `'module'` | Load a standard JavaScript module |
+| `'commonjs'` | Load a Node.js CommonJS module |
+| `'builtin'` | Load a Node.js builtin module |
 | `'dynamic'` | Use a [dynamic instantiate hook][] |
 
 For example, a dummy loader to load JavaScript restricted to browser resolution
@@ -254,5 +411,6 @@ then be called at the exact point of module evaluation order for that module
 in the import tree.
 
 [Node.js EP for ES Modules]: https://github.com/nodejs/node-eps/blob/master/002-es-modules.md
-[addons]: addons.html
 [dynamic instantiate hook]: #esm_dynamic_instantiate_hook
+[`module.createRequireFromPath()`]: modules.html#modules_module_createrequirefrompath_filename
+[ESM Minimal Kernel]: https://github.com/nodejs/modules/blob/master/doc/plan-for-new-modules-implementation.md

doc/node.1

@@ -270,6 +270,9 @@ Print stack traces for process warnings (including deprecations).
 .It Fl -track-heap-objects
 Track heap object allocations for heap snapshots.
 .
+.It Fl -type Ns = Ns Ar type
+Set the top-level module resolution type.
+.
 .It Fl -use-bundled-ca , Fl -use-openssl-ca
 Use bundled Mozilla CA store as supplied by current Node.js version or use OpenSSL's default CA store.
 The default store is selectable at build-time.