esm: The getPackageType utility function is not exposed

[DerekNonGeneric]

Is your feature request related to a problem? Please describe.
Assist in determining module format by using internal machinery.

Describe the solution you'd like
Expose the getPackageType utility function.

Describe alternatives you've considered
The alternative is what I am currently doing, which is passing the --expose-internals flag (no bueno) and acquiring the utility function via internalBinding('module_wrap'). Exposing it would allow shaving off one dangerous flag and five lines from the following example.

// Flags: --expose-internals --experimental-modules --experimental-loader

import { createRequire /*, getPackageType*/ } from 'module';
import { dirname as pathDirname } from 'path';
import { fileURLToPath as pathFromFileUrl } from 'url';

const __filename = pathFromFileUrl(import.meta.url);
const __dirname = pathDirname(__filename);
const require = createRequire(__dirname);
const { internalBinding } = require('internal/test/binding');
const { getPackageType } = internalBinding('module_wrap');

// const TYPE_NONE = 0;
// const TYPE_COMMONJS = 1;
const TYPE_MODULE = 2;

/**
 * @param {string} specifier
 * @returns {Promise<{url: string, format: string}>}
 */
export async function resolve(specifier) {
  const type = getPackageType(specifier);
  if (type === TYPE_MODULE) {
    return {
      url: specifier,
      format: 'module',
    };
  }
}

As you may have noticed, one use-case is for custom loaders, which necessitate this info. Amongst others, one advantage of using the internal machinery (over writing it in JS) is that it is implemented in C++, which is important because this particular algorithm can be very expensive to perform and ( ! ) resolve hooks are on the hot code path of every module.

Additionally, exposing this function may also improve the viability of using loaders to fill the current gap in format autodetection until the --es-module-resolution flag's proposed auto mode is ready.

cc @nodejs/modules-active-members
bcc @GeoffreyBooth (please cc above)

[GeoffreyBooth]

@nodejs/modules-active-members

[devsnek]

I'm against exposing machinery like this. We are planning to expose functions to let you call up to the previous loader.

_{I think this is also a really good example of why we shouldn't be adding new fields to package.json but oh well}

[guybedford]

Package type does not imply the module format, but rather the module format database, so note that the usage example is wrong as it doesn't support eg .cjs.

The source of truth to follow for determining the module format of a file is the specification for the resolver (https://nodejs.org/dist/latest-v13.x/docs/api/esm.html#esm_resolver_algorithm), under the function ESM_FORMAT.

Exposing or having userland implementations of the spec would be very useful indeed. If Node core were to expose something here it should likely be a JS esmFormat implementation. Then how to handle caching and whether it should be sync or async come up. Both have pros and cons. As you can see that is not necessarily trivial from a development and consensus perspective. Contributors to this process welcome! Userland work would likely be a great starting point.

[GeoffreyBooth]

Does Node already do the package.json lookup before this loader is called? Presumably it must have, I would think, to know how to reach this file (such as if it were referenced via "exports"). If so, it would both improve performance and solve this particular need if Node could pass along the package.json type (or all package.json metadata) to the loader, so that the loader doesn’t need to look the data up again.

[bmeck]

I think exposing something is likely useful here since it is possible to sniff / can be looked up on disk and there isn't a clear workaround to make it easy to deal with. The main contention here for myself is what to expose; as @guybedford points out, just exposing the "type" might not be the best choice as it doesn't really expose the underlying mechanisms that the runtime is using when that is set and could lead to misunderstanding and misuse.

[GeoffreyBooth]

To give a concrete use case, I’m considering creating a CoffeeScript loader. So for a given file.coffee that this loader would process, I need to know the package type the same as if the file was file.js (since for CoffeeScript, we can’t store the parse goal metadata in the file extension and therefore I need to know the nearest parent package.json "type").

So for me, I’d want to know either what Node would determine the package scope type to be if my file were a .js file; or I’d want Node to provide to the loader the contents of the controlling package.json file.

[devsnek]

I feel like this points to an inherent problem that tooling needs to know way too much about the specific details of node's esm resolution system. In core we've been really careful to avoid adding new stuff like that to the cjs loader because of the burden on the ecosystem to support it and the node maintainers to keep it working for the ecosystem.

even if we expose all the needed APIs, that doesn't guarantee tooling uses them, or uses them at the correct times (matching the behaviour of bugs is important too).

[bmeck]

@GeoffreyBooth that sounds like you need to know the format of a file, not the package type.

[GeoffreyBooth]

@GeoffreyBooth that sounds like you need to know the format of a file, not the package type.

What do you mean? The format is CoffeeScript.

tooling needs to know way too much

Well, why does the loader API need to be given the type? Ideally that would be optional, and if it’s unspecified the regular Node logic would apply; and unknown types (e.g. '.coffee') could be treated the same as .js, or I would have some way of telling Node to treat them like .js. And by “treat as .js” I mean, use the same logic that Node’s ESM loader uses to determine whether to treat .js as CommonJS or as ESM.

[bmeck]

@GeoffreyBooth

I’d want to know either what Node would determine the package scope type to be if my file were a .js file; or I’d want Node to provide to the loader the contents of the controlling package.json file.

As you stated, you want to know what format a .js file is, not specifically the string value of the "type" field. Anyways, "package scope type" isn't done per file; a package has a set of formats mapping to file extensions within its boundaries. So, the package scope type of a file doesn't really make sense. You can see what format a package scope would treat a file to be though.

[guybedford]

Here is an implementation for getPackageType:

function getPackageType (checkPath) {
  const rootSeparatorIndex = checkPath.indexOf(path.sep);
  let separatorIndex;
  while (
    (separatorIndex = checkPath.lastIndexOf(path.sep)) > rootSeparatorIndex
  ) {
    checkPath = checkPath.slice(0, separatorIndex);
    if (checkPath.endsWith(path.sep + 'node_modules'))
      return 'commonjs';
    try {
      const pjson = JSON.parse(fs.readFileSync(checkPath));
      return pjson.type || 'commonjs';
    }
    catch (e) {
      if (e.code === 'ENOENT') continue;
      throw e;
    }
  }
  return 'commonjs';
}

This follows the specification (which is fully provided in the Node.js documentation exactly for this reason).

Because it is specified and relatively stable, there is nothing wrong with copy-pasting such a snippet. DRY does not apply when there are lots of decisions like API / async or async / caching concerns. Copy paste is good for this stuff. Or a userland packages with this code would be useful. Please be useful.

[GeoffreyBooth]

Thanks @guybedford, that certainly solves the need; though if this needs to happen for many/most loaders we might want to consider providing this as a utility, or reusing Node’s lookup rather than having this happen twice for each file.

This also makes me think of nodejs/modules#436; I wasn’t aware we treated a node_modules folder as a special case. It shouldn’t need to be, as every subfolder of node_modules should have a package.json; and we’re not really documenting this in the main docs, just in the resolver spec. I get that this is probably to cover some weird edge cases of files in node_modules that aren’t in subfolders, or something like that, but I wonder what the harm would be in removing this special casing.

[devsnek]

folders within node_modules aren't guaranteed to have package.json, that's just a side effect of package managers. it's really important to highlight that prior to our esm loader, package.json didn't infer a "boundary" to node.

[DerekNonGeneric]

/to @guybedford

Package type does not imply the module format, but rather the module format database, so note that the usage example is wrong as it doesn't support eg .cjs.

The goal of the example provided was to support use-cases like @GeoffreyBooth's, which don't use file extensions to determine module format; sorry, I should've clarified.

This follows the specification (which is fully provided in the Node.js documentation exactly for this reason).

Although the specification in the documentation may not need GET_PACKAGE_TYPE to accomplish its goal, providing pseudocode for it might be useful to those implementing it in other languages. It looks like your implementation is a simplification of ESM_FORMAT, READ_PACKAGE_JSON, READ_PACKAGE_SCOPE.

/to @GeoffreyBooth

I get that this is probably to cover some weird edge cases of files in node_modules that aren’t in subfolders, or something like that,

I will provide an example of a project directory structure that currently uses this pattern. Since it's more relevant to the other issue I opened, I'll put it in there.

but I wonder what the harm would be in removing this special casing.

Ditto. I'm guessing that it would mean that there would need to be a package.json in there instead.

/to @devsnek

it's really important to highlight that prior to our esm loader, package.json didn't infer a "boundary" to node.

There are actually two boundaries now: package.json and node_modules.

[bmeck]

@DerekNonGeneric

The goal of the example provided was to support use-cases like @GeoffreyBooth's, which don't use file extensions to determine module format; sorry, I should've clarified.

to the comment above, @GeoffreyBooth 's use is to map to what format the .js file extension maps to and does directly relate to file extensions to my understanding. Is there a different use case that is unrelated to file extensions I've missed?

[DerekNonGeneric]

/to @bmeck

to the comment above, @GeoffreyBooth 's use is to map to what format the .js file extension maps to and does directly relate to file extensions to my understanding.

Going off of what he wrote in the following excerpt.

And by “treat as .js” I mean, use the same logic that Node’s ESM loader uses to determine whether to treat .js as CommonJS or as ESM.

A file extension-based solution is not viable because CoffeeScript files use the .coffee extension, so using the getPackageType algorithm would be the sole way of determining their module format. I'm not familiar with CoffeeScript, so please correct me if I've misinterpreted it. The example was intended to be extension-agnostic.

[bmeck]

@DerekNonGeneric I don't understand the reasoning you give.

so using the getPackageType algorithm would be the sole way of determining their module format.

This is simply not true; any exposure of the format database mapping would suffice and likely lead to clearer more robust code, e.g.

getFileFormatsForExtensions(directoryOfFileInQuestion).get('.js');

This is clearer in wanting to match .js rather than mapping a map of extension to format being used as a single format. This would also handle more use cases and be less fragile over time than keeping your own mappings of things (see concern such as the .cjs mapping above). This also has the added advantage of if new shorthands arise type: 'web' or w/e that the tooling can pick up on that mapping rather than bailing out due to an unknown string.

[DerekNonGeneric]

/to @bmeck

@DerekNonGeneric I don't understand the reasoning you give.

so using the getPackageType algorithm would be the sole way of determining their module format.

If CoffeeScript files must use the .coffee extension, they cannot use the .js, .mjs, or .cjs extensions, right? The module's extension is serving the purpose of defining the module's syntax (CoffeeScript) and can't be used for defining the module's format (CommonJS, EcmaScript).

I already proposed .cjs.coffee or .mjs.coffee if some sort of mapping were to be of any relevance.

This is simply not true; any exposure of the format database mapping would suffice and likely lead to clearer more robust code, e.g.

Is this module format database hypothetical or does it actually exist? I would like to know more.

It would be nice if it could expose an API that would enable the querying of hypothetical file URL strings.

Like:

Hey database,
if this extensionless file URL string had a `.js` extension,
what module context would you say it's in?

Similarly:

Hey database,
if this `.coffee` file URL string had a `.js` extension,
what module format would you say it is?

[bmeck]

@DerekNonGeneric

If CoffeeScript files must use the .coffee extension, they cannot use the .js, .mjs, or .cjs extensions, right? The module's extension is serving the purpose of defining the module's syntax (CoffeeScript) and can't be used for defining the module's format (CommonJS, EcmaScript).

They don't need to use an extension or even put something on disk, they just need loaders to map the location to a format (the same format as .js in the examples above). CoffeeScript is seeking to match the format of .js, I'm not sure I understand the syntax vs format terminology you are using here and the differences of them.

I already proposed .cjs.coffee or .mjs.coffee if some sort of mapping were to be of any relevance.

We should probably not encourage multi . usages as many tools don't support that (OS/stdlib/etc don't). In addition various things like data: URLs aren't made for this kind of nesting.

Is this module format database hypothetical or does it actually exist? I would like to know more.

the databases already exist within our runtime, but some design considerations about them would need to be discussed regarding "custom" mappings.

It would be nice if it could expose an API that would enable the querying of hypothetical file URL strings.

I'm not sure I understand the "hypothetical URL strings" bit. You would just query the database relative to a location. This is what node does already with package types.

[GeoffreyBooth]

I'm not sure I understand the "hypothetical URL strings" bit. You would just query the database relative to a location. This is what node does already with package types.

I think that’s just “Hey Node, if path file:///usr/src/app/file.coffee ended in .js, how would you parse it?” As in, the hypothetical part is that there’s a hypothetical .js extension.

And yes, if there was some API that Node offered that could answer a question like this, in a way that I knew what string to return for format in the loader API, that would solve the problem.

[jkrems]

The module's extension is serving the purpose of defining the module's syntax (CoffeeScript) and can't be used for defining the module's format (CommonJS, EcmaScript).

I'm not sure I understand the distinction between syntax and format this makes. CommonJS and ES modules are different in terms of syntax. They have a lot of overlap but in the end their grammar (syntax) is different. There's no such thing as a return statement in an ES module body just like there's no such thing as an export statement in CommonJS.

Some tools have parsers that accept a cover grammar over both but the end result is that they accept input that's neither. E.g. export default 0; return 20; is parsed happily by TypeScript's JS mode but is neither valid CommonJS syntax nor valid ESM syntax. CoffeeScript made a similar choice by allowing incompatible syntax in the same file which is why it happily generates invalid JS for export default 20\nreturn 0 (invalid because it's not any existing JS syntax). The fact that CoffeeScript allows export in .coffee is a symptom of it not properly splitting the grammars, not proof that ESM and CommonJS share the same syntax/grammar. :)

P.S.: Babel seems to have fixed some of the issues it had in the past. It doesn't support CommonJS but it supports script which is mostly a clean subset of CommonJS and module. Which means it doesn't allow mixing HTML comments and export statements. The return example works in neither mode because that's a CommonJS-only "feature".

[weswigham]

"parsed happily" isn't the same as "doesn't have an error that simply isn't reported". A parser should be error tolerant when used in editor scenarios...

[jkrems]

@weswigham TypeScript does have great reasons for the current behavior - there's a lot of code that assumes import is just sugar that gets compiled away into require. It would be awful if vscode would bail on that code just because it happens to not be compatible with a certain set of runtimes or tools. But it still means that these graceful cover grammars are a poor predictor of what syntax "native" runtimes would support. And it can create the impression expressed above that ESM and CJS have the same syntax.

[DerekNonGeneric]

At risk of being pedantic, here's further clarification if it's still needed.

Correction:

The module's extension is serving the purpose of defining the module's dialect and can't be used for defining the module's format.

---

dialect

• JavaScript
• CoffeeScript
• TypeScript

format

• builtin
• commonjs
• dynamic
• json
• module
• wasm

---

We cannot use file extensions to store metadata used to determine format because the various dialects mandate that source file extensions be associated with their respective dialect.

[bmeck]

We cannot use file extensions to store metadata used to determine format because the various dialects mandate that their source files have the extension associated with that dialect.

I don't understand this at all. We have counter examples above. I don't understand what dialect is being defined as specifically. Each format in the loader sense is a unique way of interpreting files, it can be a mapping to another format (such as .coffee wanting to compile to the same target as w/e .js maps to). In all the examples above we aren't associating Coffeescript with Javascript, we are a loading operation for Coffeescript with the format of .js files which could be CommonJS (application/node in MIME) or ES Modules (text/javascript in MIME) in our runtime.

[weswigham]

@bmeck for TS, at least, since commonjs is the only module target that really exists, we assume a .ts file that exists today is ultimately targeting commonjs. (As that is what it does today) For all the same reasons .mjs exists, we'll probably also need to add a .mts (and .mtsx, and .d.mts) to allow users (and the compiler itself) to distinguish between cjs-targeting and esm-targeting files in their projects. Then, hopefully, we layer on support for type: module the same way node has to allow the default behavior for .ts and company to change in a scoped way, and add support for .cts (and .ctsx and .d.ctsx) to opt back down to cjs in those scopes. Do note that this requires adding at least six new extensions to lookup on disk for us (which is roughly double what we check today). If we were to leverage some built-in functionality in node to accomplish this, it would need to be general enough to allow associating these arbitrary extensions with arbitrary builtin formats at custom priorities, while performing the resolution operation. Indeed, it may just be better to expose something like getPackageScopedSetting(path: string, fieldname: string) which just returns the closest package up the spine with that setting set and its value (as is sorta used for looking up the type field internally), and leave the actual resolution augmenting to tools reimplementing resolution.

[bmeck]

Indeed, it may just be better to expose something like getPackageScopedSetting(path: string, fieldname: string) which just returns the closest package up the spine with that setting set and its value (as is sorta used for looking up the type field internally), and leave the actual resolution augmenting to tools reimplementing resolution.

I disagree; even with the multitude of file extensions being a possible UX that TS chooses to take, it would still ultimately be trying to understand the formats within the package scope. Nothing we have discussed prevents customization of these and there have been multiple discussions on allowing augmentation to be explicit and in package.json such that tools can coordinate properly. If TS were to implement its own augmentation any Node loaders for TS would need a way to be aware of these (either by hard coding, or coordination) and the same would be true for other tooling like linters or build tooling. Leaving it up to tooling doesn't simplify things, and the argument that tools might want to associate their own mappings is not prevented by exposing mappings. If we consider real world examples of things like how eslint is being forced to work around how to load files due to not knowing the formats at a location and/or the format of files being misaligned with package.json it seems that a need is being left unfulfilled, not that we should leave all tooling to find their own workaround.

[DerekNonGeneric]

Using require.resolve(request[, options)] as a template, here's an idea that may fill the needs expressed.

---

import.meta.resolve(request[, options])

• request <string> The module path to resolve (specifier).

• options <Object>

  • paths <string[]> Paths to resolve module location from. Each of these paths is used as a starting point for the module resolution algorithm.
  • ext <string> The inferred extension of the provided request. Used to override the actual file extension if present or to define one for an extensionless path. Default is .js if extensionless.
  • boundaries <string[]> Package boundaries. Used to override the default package boundaries. Default is ['node_modules/', 'package.json'].

• Returns: <Object>

  • url <string> The resolved file URL string.
  • format <string> The resolved module format.

Use the internal esmFormat() machinery to look up the location and format of a module, but rather than loading the module, just return the resolution object.

---

A note I'd like to make is that if this function were internally implemented in userland JS (as require.resolve is), it would likely provide no performance advantage over such a function existing in one's own custom loader.

[GeoffreyBooth]

I just played around with the resolve hook a bit. The third parameter is the default ES resolver. I tried writing a loader like this:

export async function resolve(specifier, parentModuleUrl, defaultResolver) {
  console.log(defaultResolver(specifier));

And I ran it via node --experimental-loader ./loader.js test/index.coffee, where the current working folder contained a package.json with "type": "module". It printed this (url shortened):

{
  url: 'file:///app/loader/test/index.coffee',
  format: 'module'
}

I put a package.json with {"type": "commonjs"} in test/ and ran it again, and got the same output but with format: 'commonjs'. This would seem to solve the need at the top of this thread. I guess the default ES resolver treats unknown extensions the same as .js for the purposes of determining format/type?

Rewriting the specifier to something that doesn’t exist, like replacing .coffee with .js, throws ERR_MODULE_NOT_FOUND; so if there’s a use case for querying the type of an existing path to a nonexistent file, then that use case is still unsatisfied.

[guybedford]

@GeoffreyBooth it seems you've found a bug actually. I've posted #30632. Following the specification, there is supposed to be an unsupported module format error thrown for this case when "type": "module" is set. It is supposed to support format: 'commonjs' in the non module case though.

[GeoffreyBooth]

Okay. Well as far as I understand this resolve hook, it’s supposed to return an object like:

{
  url: 'file:///app/loader/test/index.coffee',
  format: 'module'
}

Could something in this return object tell the loader to do the “determine the package type as if url was a .js file” logic? Like leaving out format, or setting it to auto or some other special string?

That way we wouldn’t need to expose this utility method, and potentially cause this logic to run twice for every specifier, but custom loaders could still take advantage of this internal method.

[jkrems]

Also, it's problematic that a resolve hook would return a format. It assumes that at URL resolution time the content-type behind a URL is already known. I would hope that a final loader hook API only returns that information when actually fetching the resource contents.

[GeoffreyBooth]

I would hope that a final loader hook API only returns that information when actually fetching the resource contents.

Speaking of which, where's the hook for transforming the source? Is that just not implemented yet? @bmeck @jkrems @a-lxe

[DerekNonGeneric]

/to @jkrems

Also, it's problematic that a resolve hook would return a format. It assumes that at URL resolution time the content-type behind a URL is already known. I would hope that a final loader hook API only returns that information when actually fetching the resource contents.

I second this hope and I wonder if we share the same concern. With the current API, if the loader needs any of the source contents to determine the module format (e.g. @module JSDoc and containsModuleSyntax), it will end up reading file contents twice (once when doing the parsing of the format in the loader, and another when being added to the cache internally).

/to @GeoffreyBooth

Speaking of which, where's the hook for transforming the source? Is that just not implemented yet?

It seems like the dynamic instantiate hook can be leveraged for that purpose. No guarantees, but I'll try to put together a demo as a proof of concept. I've been looking for a reason to get familiar with this hook. Please let me know if you find success before I do. 😃

/to @guybedford

If Node core were to expose something here it should likely be a JS esmFormat implementation. [...] Contributors to this process welcome! Userland work would likely be a great starting point.

Here are the three functions associated with esmFormat implemented in userland JS. They have the specification inlined as comments and have been typechecked with the Closure and TypeScript compilers. Sorry for the delay, I wanted to use them for a bit to make sure they are working as expected.

Userland JS implementation of Node's ESM resolver spec

I will also be doing the remaining three and write unit tests for them. Hope this helps! I must know how we can better collaborate. Where would these go in the Node core repo? I need more guidance about contributing things like this.

[guybedford]

Absolutely amazing work to see @DerekNonGeneric! We discussed having an open test suite for the resolver at the last meeting, if you want to work with core to get the unit tests made as a suite that can be tested against userland resolvers as well that could be an interesting effort, but not pressure on that either.

Just having those functions available to users spec compatible as a resource to install or fork in their own apps unlocks a huge amount of value.

[bmeck]

I'd still like to try and decouple determining format/metadata from determining location. There are a few ways we could do that, but if we do go with an API like loaderMetaData(url: String<URL>) I'd want the return type to be extensible and to ensure the url does not need to exist. Such an seems a bit more than the original discussion of this issue though and perhaps it would be good to expose a different issue for each of these APIs and a tracking/coordination issue. The original issue here to me at least seems to be solved by loaderFileFormatMappingWithin(url: DirName<String<URL>>) that fails for non-file: URL schemes.

[DerekNonGeneric]

The original issue here to me at least seems to be solved by loaderFileFormatMappingWithin(url: DirName<String<URL>>) that fails for non-file: URL schemes.

@bmeck, did you mean to say would be solved? (Unclear if you're proposing a solution or stating that one already exists.)

[guybedford]

I've removed the modules agenda label here as this has been discussed a number of times now. If anyone wants to re-add though feel free.

[DerekNonGeneric]

@guybedford, what was the conclusion?

[GeoffreyBooth]

We just merged #30986 which I think makes the need for this function unnecessary. Within resolve you no longer need to return format, and inside getFormat you can do this:

// Ask Node to determine the format as if the current URL had a .js extension
return defaultGetFormat(`${url}.js`);

[DerekNonGeneric]

@GeoffreyBooth, sounds good to me.

[DerekNonGeneric]

Build tooling in need of this feature, please refer to #49446.