File extension/directory index resolution in ESM

[GeoffreyBooth]

The topic of file extension and directory index resolution (whether automatic or opt-in) is on the agenda for this week, but I can’t seem to find an issue where it has been discussed on its own. It’s not one of the features on the list in our README, though it was mentioned in our road map as being a feature removed as part of Phase 1. It was mentioned by @devsnek and @ljharb in #253 as features that they would want before the new implementation is released. I thought this topic should get its own issue so that the broader group’s attention is drawn to it, since the entire group is significantly larger than the people who attend the meetings.

I’m going to try to keep this post neutral, and I’ll update this top post as needed when people point out things I haven’t thought of.

What file extension/directory index resolution is

In CommonJS, it’s idiomatic to write code like this:

require('./startup');

Where ./startup is resolved as either ./startup.js or ./startup/index.js, or potentially other file extensions.

Why users like it

Aside from saving users from typing three or nine characters, there’s a big usability benefit in saving refactoring effort when a file grows to the point where it needs to be refactored from one into several. In the example above, your project could start with a startup.js file and when it grows unwieldy you could refactor into startup/index.js which in turn requires several other files, and you wouldn’t need to update any of the files that already require('./startup').

What we need to decide

Since we’re planning on building loaders, and since it seems fair to assume that loaders will be capable of rewriting specifiers, file extension/directory index resolution will be possible one way or another in ESM. So the question isn’t whether it should be possible, but rather how easy it should be: enabled by default, enabled via opt-in package.json configuration, or enabled via a package-level loader. Or looked at another way, do we want to encourage the practice, slightly discourage it, or strongly discourage it? The arguments for not enabling it automatically are usually the same, so I’m grouping those together.

Arguments for supporting file extension/directory index resolution automatically

• This is already supported automatically by require in CommonJS. That’s what users are familiar with.

• It’s idiomatic to leave off extensions in CommonJS; actually typing them is unusual.

• It’s also idiomatic to leave off extensions in ESM JavaScript written to be transpiled by Babel or esm. There are countless projects out there written in ESM syntax that assume automatic file extension/directory index resolution, and many of those projects might very well work out-of-the-box if we continued to support automatic file extension/directory index resolution.

Arguments against supporting file extension/directory index resolution automatically

• Browsers don’t automatically resolve file extensions or directory index files (unless a server is explicitly configured to do so, but this is practically unheard of), and we’re aiming to be as equivalent with browsers as possible. As the import maps proposal puts it:

  Note how unlike some Node.js usages, we include the ending .js here. File extensions are required in browsers; unlike in Node, we do not have the luxury of trying multiple file extensions until we find a good match. Fortunately, including file extensions also works in Node.js; that is, if everyone uses file extensions for submodules, their code will work in both environments. . . .

  Unlike in Node.js, in the browser we don’t have the luxury of a reasonably-fast file system that we can crawl looking for modules. Thus, we cannot implement the Node module resolution algorithm directly; it would require performing multiple server round-trips for every import statement, wasting bandwidth and time as we continue to get 404s. We need to ensure that every import statement causes only one HTTP request; this necessitates some measure of precomputation.

  By making file extension/directory index resolution opt-in, we encourage people to write packages that are compatible with browsers by default. This is especially important because leaving off the extension is currently idiomatic; we’re encouraging the establishment of a new idiom, where code is idiomatically capable of running in either environment (at least, with regards to this; obviously there are lots of other ways people can create packages that are incapable of running in browsers).

• Node may someday support import of network path URLs, like import Three from 'https://unpkg.com/three@0.102.1/src/Three.js'. Presumably extension searching will not apply to such URLs. It would be inconsistent for automatic file extension/directory index resolution to work for file:// URLs but not https:// URLs. Such a system would mean that a package relying on automatic file extension/directory index resolution would work when installed locally but fail when loaded from a network URL.

• There’s a performance cost to automatic file extension/directory index resolution, that continues to grow as more supported file extensions are added. It’s one thing to search for startup.js, and then when it’s not found to search for startup/index.js; it’s another to search for startup.js, startup.mjs, startup.wasm (etc.) and then startup/index.js, startup/index.mjs, startup/index.wasm etc. This performance hit is a big reason that require.extensions was deprecated. From Node’s docs:

  Note that the number of file system operations that the module system has to perform in order to resolve a require(…) statement to a filename scales linearly with the number of registered extensions. In other words, adding extensions slows down the module loader and should be discouraged.

  By requiring this to be opt-in via configuration or a loader, that configuration or loader can specify how this feature should behave. For example, maybe .mjs is the only supported extension automatically resolved, and the performance cost is reduced.

• The UX becomes more complicated now that we have multiple supported extensions. When a folder contains file.js, file.mjs and file.wasm and you import ‘./file’, which file gets loaded? When support for new extensions is added to core, do they always get added last in precedence?

• We can release our new implementation unflagged without automatic file extension/directory index resolution and choose to add it later, but we can’t do the reverse.

Arguments for supporting it via package-level configuration

• As part of Node proper, performance would be as optimized as it could be.

• This would standardize users’ use of file extension/directory index resolution in a way that supporting it via third-party loaders would not.

Arguments for supporting it via a package-level loader but not via configuration

• This would send a strong signal from Node that leaving off extensions is discouraged. Getting the feature would require either the performance hit of adding a loader, or the complexity of transpilation/building.

If people offer new arguments (or correct these) I’ll update this post.

Background: https://youtu.be/M3BM9TB-8yA?t=835

[devsnek]

Thank you for opening this issue and listing everything out 😄

I'd like to pick the option that obstructs the fewest use cases. Additionally, I don't think that node should push things like browser conformance on users, given that it isn't a browser and that a huge section of the ecosystem feels this is a useful and valid feature, that that it's been part of the ecosystem for ten years.

It is absolutely valid to want to continue to use resolution, and if resolution exists by default it doesn't stop those who don't want to use it from not using it. If you put full filenames in, you can be browser compatible and avoid what you judge to be performance issues, all without needlessly blocking other people's preferences.

My point in all the above is that I don't think either side of this disagreement is hurt by leaving this in. I'd be very open to adding options to disable the feature, but I really think everyone gets what exactly what they want by leaving it on by default. I'm hopeful we can compromise :)

[robpalme]

I think the primary argument in favour of extension-probing is continuity with the old module system. It preserves those conventions.

The main consequence of this decision will be to influence the conventions used by the modules written and published to npm after we release/unflag. Defaults are very powerful, so we need to ask ourselves not "what did we do before?" but "what do we want to encourage in the future?".

On this topic, my opinion is that we ought to promote convergence with Browsers (and other emergent JS platforms). I accept nothing compels us to strive for this, and this is a subjective viewpoint, and that others are happy for Node and Browser to diverge.

We've covered this extension-probing a few times over the years without killer-arguments and without concensus. Maybe we're due a binding vote on this? I think a decision (any decision) will be healthy for the working group and the ecosystem.

[GeoffreyBooth]

Maybe we're due a binding vote on this? I think a decision (any decision) will be healthy for the working group and the ecosystem.

Shipping ESM without automatic resolution isn't necessarily making a decision to not support automatic resolution in ESM ever; support can always be added at any time via PR. Indeed that might be preferable, as then a much broader constituency can debate the merits of encouraging browser compatibility versus providing the convenience of automatic resolution.

In contrast, shipping with automatic resolution would be making a decision, as later removing it would mean a breaking change that would affect a lot of users. ESM would be stuck with it forever like CommonJS is.

[ljharb]

Yes, it is - shipping it later is a breaking change. For defaults, at least, we largely have only one shot at this - everything we want needs to be included at the outset, or it likely won’t ever be able to be added.

I’ll also point out; without extension resolution, coffeescript could never have taken off, as migrating a file to or from coffeescript would have been a breaking change. CJS isn’t “stuck” with it, it has massively benefited from it, and continues to do so.

[zenparsing]

Thanks @GeoffreyBooth for this review. The only thing I would add is that I found the two questions presented here to be highly coupled: if we are going to allow directory imports, then we will probably want to do so using some kind of index plus file extension search, as with require. (The other option would be to use/abuse package.json for directory imports.) If we decided to support file extension lookup for directory imports, then it we might as well support it for non-directory lookups as well.

[MylesBorins]

@ljharb can you please expand on why adding it later is a breaking change?

…

On Tue, Feb 19, 2019, 12:30 PM Kevin Smith ***@***.***> wrote: Thanks @GeoffreyBooth <https://github.com/GeoffreyBooth> for this review. The only thing I would add is that I found the two questions presented here to be highly coupled: if we are going to allow directory imports, then we will probably want to do so using some kind of index plus file extension search, as with require. (The other option would be to use/abuse package.json for directory imports.) If we decided to support file extension lookup for directory imports, then it we might as well support it for non-directory lookups as well. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#268 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAecV-xmD6_r6dy51qmDiogJHdc5ZYAaks5vPDSbgaJpZM4bCUDW> .

[ljharb]

With extension lookup matching cjs, an extensionless file next to an extensioned file of the same name matches the extensioned version first; without it, it matches only the extensionless file. Adding extension lookup breaks this case, unless I’m misunderstanding something.

We could of course deviate from how cjs does it and be nonbreaking, but that wipes out many of the reasons to do it (altho not all).

[jkrems]

@ljharb You mean require('./foo') where a literal 'foo' exists? I'm pretty sure that would take the literal 'foo' first:

> echo 'console.log("foo");' > foo

> echo 'console.log("foo.js");' > foo.js

> node -e 'require("./foo")'
foo

[GeoffreyBooth]

With extension lookup matching cjs, an extensionless file next to an extensioned file of the same name matches the extensioned version first; without it, it matches only the extensionless file. Adding extension lookup breaks this case, unless I’m misunderstanding something.

As far as I’m aware, and someone please correct me if I’m wrong: In ESM, import only works for recognized extensions. You can’t import an extensionless file.

This was done to avoid breaking changes as new extensions are added in the future. For example in CommonJS I think require('./file.wasm') today imports file.wasm as CommonJS JavaScript, the same as if it were file.foo, which is a problem whenever we add support for WASM someday. In ESM, unrecognized extensions throw so that we can add extensions like .wasm in the future without those additions being breaking changes.

So import './file' in the current implementation should throw, even if an extensionless file named file exists. Therefore we can add automatic resolution later and it’s not a breaking change, any more than adding new syntax causes previously throwing things to no longer error.

Edit: I’m referring to ESM mode in our current new implementation, not the ESM spec itself.

[bmeck]

As far as I’m aware, and someone please correct me if I’m wrong: In ESM, import only works for recognized extensions. You can’t import an extensionless file.

This is not true for ESM, this is just how the stripped down loader works. ESM itself doesn't mandate anything about specifier string format.

[GeoffreyBooth]

This is not true for ESM, this is just how the stripped down loader works. ESM itself doesn’t mandate anything about specifier string format.

Sorry, I guess I should’ve said ESM in our implementation. Clearly in the browser, browsers don’t care about extensions. Is what I wrote true for our implementation?

[bmeck]

@GeoffreyBooth for the minimal kernel yes, but not for the current upstream.