Skip to content
This repository has been archived by the owner on Sep 2, 2023. It is now read-only.

Feedback on extension resolution #323

Open
ljharb opened this issue Apr 28, 2019 · 158 comments
Open

Feedback on extension resolution #323

ljharb opened this issue Apr 28, 2019 · 158 comments
Labels
cjs features surveys Relates to things where people to you what you don't want to but need to hear

Comments

@ljharb
Copy link
Member

ljharb commented Apr 28, 2019

It seems like a good idea to capture somewhere the feedback we receive on extension resolution.

@ljharb
Copy link
Member Author

ljharb commented Apr 28, 2019

nodejs/node#27407

@ljharb
Copy link
Member Author

ljharb commented Apr 28, 2019

nodejs/node#27408

@devsnek devsnek added cjs features modules-agenda To be discussed in a meeting surveys Relates to things where people to you what you don't want to but need to hear labels Apr 28, 2019
@AlexanderOMara
Copy link

Is there any information available on why file extension resolution was disabled?

@GeoffreyBooth
Copy link
Member

GeoffreyBooth commented May 1, 2019

@sheerun

This comment has been minimized.

@AlexanderOMara
Copy link

AlexanderOMara commented May 7, 2019

UPDATE: I think this is actually the biggest problem: #323 (comment) , #352

The below, while not super pleasant, could be worked around with smarter transpilers relatively easily.


Before, it was possible to write ES modules (or TypeScript modules), and publish both CommonJS and ES modules with a simple module transformation via babel (I've actually started doing this already, so my modules can be treeshaken).

Now we can't write:

import {foo} from './bar';
console.log(foo);

We have to write:

import {foo} from './bar.mjs';
console.log(foo);

Which means that now any transpiler also needs to rewrite the actual import path. I don't think any currently have this functionality, because until now this wasn't an issue.

This is very different from how the ecosystem has worked so far. For example, TypeScript and Webpack both work just fine with the extension-less module resolution. I would guess rollup does too.


I thought the previous design made much more sense.

When I import something, naturally I expect to get an ES module, preferring the .mjs extension, probably falling back on a .js CommonJS module exported as an ES module.

If I require something, naturally I expect to get a CommonJS module, preferring the .js extension which can't really be changed for backwards compatibility, probably falling back on loading a .mjs ES module exported as a CommonJS module.

I get that some people don't like the new file extension, but I don't think it makes sense to create new problems and add ambiguity just to keep it.

@MylesBorins
Copy link
Contributor

Some feedback in favor of not having extension searching on by default

Twitter poll about compat: 65% of 1295 people favor Browser compat

@jhnns on twitter: https://twitter.com/Jhnnns/status/1003201464716726272

@mhart on twitter: https://twitter.com/hichaelmart/status/1039529625100185600
@brianleroux on twitter: https://twitter.com/brianleroux/status/1039653429272952832

Announcement tweet for PR had 150k impressions and no negative feedback

Tweet asking for feedback

Obviously none of this is scientific... but thought I could offer some balance to the feedback

@ljharb
Copy link
Member Author

ljharb commented May 15, 2019

Again tho, enabling extension resolution does not in any way prevent “browser compat” - this is purely a question of whether you want the preferences of one group (browser compat folks) to oppress another (back compat folks) by making the feature off by default - because on by default causes no damage, but off by default does.

@AlexanderOMara
Copy link

Is browser compatibility really even possible? As soon as you import another module, isn't browser compatibility lost?

import {something} from 'some-module';
console.log(something);

In node, that would have to resolve to something in node_modules right? You're not expected to write import {something} from 'node_modules/some-module/index.mjs'; right?

Am I missing something here?

@jkrems
Copy link
Contributor

jkrems commented May 15, 2019

@AlexanderOMara Some browsers have shipped support for import maps which do make the code above work, assuming the appropriate import map is provided that tells the browser where some-module lives.

@MylesBorins
Copy link
Contributor

@ljharb I disagree with your claim that turning it on by default has no undesirable outcomes. I also think we can avoid boxing people into categories and creating an us vs them narrative. We all care about our ecosystem and making a great developer experience.

@AlexanderOMara the import map proposal

Chrome platform status: https://www.chromestatus.com/feature/5315286962012160
Tracking issue: https://bugs.chromium.org/p/chromium/issues/detail?id=848607

@AlexanderOMara
Copy link

I think it would help a lot if the proposed changes to the ecosystem were better fleshed out and documented.

  • From a package author's perspective, what are the implications?
  • From a transpiler's or to-JS compiler's perspective, what new features are needed?
  • From a user's perspective, what are the advantages and disadvantages of this change?

I don't think the changes are just about browser-support vs backwards-compatibility. As I understand it, there would also be a loss of certain functionality, but potentially also a gain of new functionality.

At this point, I'm honestly not sure which system I would prefer once all the pieces are in place, and currently I think it's a lot easier to see the negative implications of this change.

@GeoffreyBooth
Copy link
Member

Let’s be clear: we shipped automatic extension resolution in ESM. It’s --es-module-specifier-resolution=node. It’s just not on by default. The people asking for extension searching could simply be told to use that flag to turn it on. It’s an inconvenience, sure, but fairly minor in my opinion. The question is whether that inconvenience is justified by what’s gained from making searching disabled by default.

As far as I can tell, to @AlexanderOMara’s point, the biggest consequence of having searching disabled by default is that public package authors can’t assume its availability—which is a big benefit, in the eyes of many in this group, as that encourages packages published to public registries to be cross-compatible with browsers by default (at least as far as resolution is concerned). That’s what would be lost if the default is flipped. To me, that benefit outweighs the cost of users needing to use a flag if they want this enabled in their projects.

From a package author’s perspective, what are the implications?

They need to publish their packages with import specifiers that include file extensions. This will annoy some authors. However, such packages will work in browsers without extra transpilation or building, assuming they aren’t otherwise incompatible (like by importing native modules or Node APIs that can’t run in the browser). Even if a particular package author is exclusively targeting Node, all the package authors who are targeting browsers (and/or Node) benefit from a broader ecosystem of more cross-compatible packages, as that expands the number of potential dependencies that a browser-targeting package can import.

From a transpiler’s or to-JS compiler’s perspective, what new features are needed?

None. Transpilers can add the ability to add extensions at compile time, so a specifier like './file' can be rewritten to './file.js' during compilation along with whatever else is getting converted by the transpiler. This would provide similar UX without needing the flag, if the user didn’t want to either just type the extension or use the flag.

From a user’s perspective, what are the advantages and disadvantages of this change?

See above. They need to use a flag if they like this behavior, so that’s potentially a disadvantage. The advantage is that if they use any publicly published packages in a browser context, those packages are easier to work with in that they don’t require a build process or a specially configured server.

@ljharb
Copy link
Member Author

ljharb commented May 15, 2019

They're already not cross-compatible with browsers because they use bare imports. import maps work the same way with bare imports as with extensions.

In other words, unless we ban bare specifiers altogether, "work in browsers without extra transpilation or building" either is a) identically true with or without default extension resolution, or b) is identically false with or without it.

@AlexanderOMara
Copy link

From a user’s perspective, what are the advantages and disadvantages of this change?

See above. They need to use a flag if they like this behavior, so that’s potentially a disadvantage. The advantage is that if they use any publicly published packages in a browser context, those packages are easier to work with in that they don’t require a build process or a specially configured server.

Suppose I'm making a module that's explicitly node-only. What do I have to do to load a native module? Do users have to add a flag just to use it?

Suppose I'm making a module that has a WASM component. Does that mean using node-only functionality, or is there a way to do that in a browser-compatible way?

@bmeck
Copy link
Member

bmeck commented May 15, 2019

Suppose I'm making a module that's explicitly node-only. What do I have to do to load a native module? Do users have to add a flag just to use it?

In general the ability to interpret different formats are not affected by resolution, so thats out of scope of this particular issue. However, the list of supported module formats could differ between Node and the browser and both have ways to intercept requests and translate to a different supported format (though I doubt the cost of doing so in the browser is worth it).

Suppose I'm making a module that has a WASM component. Does that mean using node-only functionality, or is there a way to do that in a browser-compatible way?

WASM requires all APIs be passed in via imports, I don't understand the question. Both browsers and Node are looking at supporting loading WASM format resources. This also seems out of scope of this issue.

@GeoffreyBooth
Copy link
Member

Re import maps, see https://github.com/WICG/import-maps#extension-less-imports:

It is also common in the Node.js ecosystem to import files without including the extension. We do not have the luxury of trying multiple file extensions until we find a good match. However, we can emulate something similar by using an import map. For example,

 {
   "imports": {
     "lodash": "/node_modules/lodash-es/lodash.js",
     "lodash/": "/node_modules/lodash-es/",
     "lodash/fp": "/node_modules/lodash-es/fp.js",
   }
 }

would allow not only import fp from "lodash/fp.js", but also allow import fp from "loadsh/fp".

Although this example shows how it is possible to allow extension-less imports with import maps, it’s not necessarily desirable. Doing so bloats the import map, and makes the package’s interface less simple—both for humans and for tooling.

This bloat is especially problematic if you need to allow extension-less imports within a package. In that case you will need an import map entry for every file in the package, not just the top-level entry points. For example, to allow import "./fp" from within the /node_modules/lodash-es/lodash.js file, you would need an import entry mapping /node_modules/lodash-es/fp to /node_modules/lodash-es/fp.js. Now imagine repeating this for every file referenced without an extension.

As such, we recommend caution when employing patterns like this in your import maps, or writing modules. It will be simpler for the ecosystem if we don’t rely on import maps to patch up file-extension related mismatches.

@AlexanderOMara
Copy link

Suppose I'm making a module that's explicitly node-only. What do I have to do to load a native module? Do users have to add a flag just to use it?

In general the ability to interpret different formats are not affected by resolution, so thats out of scope of this particular issue.

I had thought the ability to import things that aren't JS modules was part of this issue. Maybe I was mistaken.

@ljharb
Copy link
Member Author

ljharb commented May 15, 2019

@GeoffreyBooth sure. and i think it's a mistake that the import maps proposal isn't yet providing a more flexible means to handle that use case - but that's not something that should make node constrain itself.

@bmeck
Copy link
Member

bmeck commented May 15, 2019

@ljharb I do not see this as a constraint, but rather a useful step in iterating so that we can ensure the end result of our process is what we desire step by step. I think going conservative with our ecosystem forwards compatibility is better than diverging for backwards compatibility. This is an issue about feedback but saying we are being constrained makes it sound like feedback about why extension resolution is useful wouldn't be accepted. We can iterate and improve our modules implementation over time :).

@MylesBorins
Copy link
Contributor

@AlexanderOMara importing things that are not js is not the issue. We have experimental support for JSON and a PR is open for WASM. We are also doing work with browser vendors and the wasm spec authors to standardize the same modules for browsers 🎉.

The bigger issue is that browsers will never do multiple network calls to resolve a file extension, so having specifiers throughout source text without file extensions creates a universe of "node only" code. It could be transpiled, but why should it have to be? Yes, people can always choose to write code with a subset if they wish to, but my feelings on the matter is that we will over time have a more stable cross platform ecosystem if we minimize the places where things diverge.

If you choose to use Node.js apis, you will find yourself in a place where your diverge... but it is possible to polyfill those specific APIS (see browserify).

@ljharb
Copy link
Member Author

ljharb commented May 16, 2019

@bmeck because of extensionless flies and type module, I’m not convinced that it won’t be a breaking change to add default extension resolution later.

The module system is not like other things; it can’t really be iterated on over time. Just like cjs, whatever we initially ship may be effectively the entirety of the system for the foreseeable future.

@bmeck
Copy link
Member

bmeck commented May 16, 2019

@ljharb

@bmeck because of extensionless flies and type module, I’m not convinced that it won’t be a breaking change to add default extension resolution later.

If this is considered breaking, it just means people could opt-in via a flag, like package.json or w/e.

However, we can be more concrete here with an example. The addition of an extension-less file in a place that it collides requires an invalid specifier:

/foo
/foo.js

import '/foo' would still find /foo before /foo.js with current algorithm for extension searching. Meaning the confusing case would actually be import '/foo' with:

/foo.js

Which would error without resolution. Moving from error to non-error seems ok to me and often is not considered breaking when adding features (particularly when some feature is missing like an API).

The module system is not like other things; it can’t really be iterated on over time. Just like cjs, whatever we initially ship may be effectively the entirety of the system for the foreseeable future.

I disagree, CJS has evolved (slowly) over time. ESM can do the same. Whatever initially ships might want to setup forward compatibility paths if desirable, but has no clear reason it is unable to match CJS' evolution over time. Part of the problem with CJS is the outstanding number of features and dynamic behavior that it exposes publicly, and we are not looking to duplicate that to my knowledge in the ESM implementation. If there are clear paths we want to reserve, we can do so; however, without clear reasons conservative iteration seems a good path forward.

@guybedford
Copy link
Contributor

@weswigham my comment wasn't entirely focused on you but more to anyone here who feels they have been sidelined in this process. Specifically what I take issue with is the claim of being left powerless while at the same time taking no action or making no contributions towards the goals of the group.

It would have been far more efficient and less of a waste of time for this not to have been an open process. This was very specifically was a fully open process though, which is an absolutely tremendous thing to have accomplished.

I don't think pursuing any directions, even when against what might seem like a majority, would ever lead to lost social capital here. I'm personally glad you've been part of the process to date and that despite your frustrations you have remained in attendance. I'm also really sorry if you feel that the process hasn't met your ideals for what you were looking to accomplish. The things that likely to lead to lost social capital are unprofessional tone, personal attacks, attempts at strong-arming or being unable to compromise on deadlocks. Blaming the process is certainly not a good look though, and if there's a specific problem with the process - then work to change that in the process!

If you really want extension searching, the onus is on you to create a proposal and add an agenda item. The ideal timeframe for that would have been a few months ago, so that this point the bar is very high indeed to get such changes through. It would likely have a few blockers (yes likely including myself), but nothing a vote couldn't ultimately overrule with enough consensus building between those who care about it. Extension searching has always very clearly been a feature that would require a vote either way, so if you want to bring such a thing back to the table now is the last call certainly.

@jkrems
Copy link
Contributor

jkrems commented Aug 4, 2020

Extension searching has always very clearly been a feature that would require a vote either way, so if you want to bring such a thing back to the table now is the last call certainly.

I think what @weswigham rightfully called out is that by keeping extension searching out of the minimal implementation, there was a de-facto bias towards not having it. So there were two positions in the working group: Having browser-like behavior and having require-like behavior. And only a vote should have resolved that impasse. But we went with a way where the browser-like behavior "won" by default, without ever needing to gather support in a vote.

What would be fair, I think, and also consistent with past discussions (IIRC), would be that the --experimental-specifier-resolution flag needs to be cleaned up. And the onus for that should fall onto the people who benefitted from the default answer "back then" - the browser-resolution camp (myself included). And it may require a vote if the current consensus doesn't support committing to the status-quo.

@weswigham
Copy link
Contributor

Having browser-like behavior

Of note, and called out way upthread, browser-like specifier behavior is pretty much out the window at this point, what with export/import map support. The only leg left standing in this area is "the complexity of tools designed to translate from package.json export/import maps to browser import maps", which is a mightily specific thing to optimize the entire specifier resolution system for, IMO, and I'm really not sure the complexity of the internals of one or a handful of tools are worth subverting years of existing expectations on all potential code.

@jkrems
Copy link
Contributor

jkrems commented Aug 4, 2020

Of note, and called out way upthread, browser-like specifier behavior is pretty much out the window at this point, what with export/import map support.

I meant it as a short-hand for relative and absolute ("non-bare") URLs, specifically. There are currently no rules for resolving bare specifiers in the browser, so that's kind of out of scope (import maps aren't a final standard yet). And for relative and absolute ("non-bare") URLs, it's very much exactly what the browser does. Unless I'm missing some piece there?

@ljharb
Copy link
Member Author

ljharb commented Aug 4, 2020

The browser, absent import maps, doesn’t support bare specifiers at all, only URLs - and node’s resolution only overlaps with data URLs, and arguably relative file URLs (which are really paths, not URLs, despite file: being a URL protocol).

I very much agree that “browser-like” is never going to be fully obtainable unless a more fully featured proposal than the current import maps proposal ends up landing in browsers; and i strongly agree with wes’ comments above. What should have required consensus was deviating from require, and it’s highly unfortunate that those of us advocating for require-like behavior got put into a position where we were scary bad blockers, and where those advocating for no extension lookup were the “good guys” who just want to see ESM shipped.

This issue definitely shouldn’t be closed.

@weswigham
Copy link
Contributor

I meant it as a short-hand for relative and absolute ("non-bare") URLs, specifically. There are currently no rules for resolving bare specifiers in the browser,

The distinction doesn't matter as far as what tooling is needed to get an arbitrary node module running in the browser. If you know and control the source, then you can include all extensions and have a crossplat library (and could have done so even if extension searching was enabled), but if you don't, the distinction doesn't matter, as your libraries may use export maps, which will need tools to modify/bundle the maps and/or code (the complexity of which is largely immaterial, as those tools aren't every-library-on-the-platform).

Additionally, I keep saying this, but it's not specified in the browser for relative specifiers - the browser just fetches whatever resource is provided at the URL by the webserver, extension or no (and provides an Accepted content type to the server). Take unpkg's webserver for instance - the webserver is clearly configured to return a specific js file for https://unpkg.com/react. I get that IIS only does default lookups for, like index.php and index.html by default, but it's configurable to do the same for js, too. Heck, that's part of the reason index.js is a thing! My point being, the browser doesn't require extensions either; it's not a given that a relative import will have an extension! Yes, a dumb static file server will just serve things as they are, but that's not the only way things work! You can't just say "browsers require extensions" because it's patently false. You can't even say "webservers require extensions" because that's not true either; at best you can argue something like "common webhosts serve files only at extensions" and my argument is simply "why does node's default dx need to cater to the least common denominator of webhosts?". Why can node, since it is a specialized runtime for running js code, not provide conveniences for the developers writing that code, should they choose to use them?

@weswigham
Copy link
Contributor

weswigham commented Aug 4, 2020

Like, node requiring extensions on relative imports is strictly less capable than what an arbitrary browser/webserver combo is capable of!

@GeoffreyBooth
Copy link
Member

It seems to me that the real failure here wasn't in our inability to reach a consensus on this issue, as it's clear that there are diverging opinions that simply won't be reconciled; our failure was in not coming up with a definitive way to resolve this question. We created the --es-module-specifier-resolution flag with the expectation that user feedback would be clear enough and plentiful enough to tell us which way to go; but I think it's safe to say that that hasn't happened. So we're simply left at a stalemate, with the automatic-resolution proponents feeling embittered because they felt that they compromised in letting the release go forward without their preferred implementation, and then that ended up being apparently the final design as there's been no clear signal from the public that the will of our users matches what shipped.

Based on comments on this thread and elsewhere, and based on the one vote we took regarding unflagging without automatic extension resolution a few years ago, it seems clear to me that if we had taken this to a vote, the current extensions-required implementation would have prevailed in a vote within the modules group. And the lack of a deluge of issues on the Node repo implies to me that the user base isn't too upset by the current implementation. So even if we didn't get here through a process we can all be happy with, it seems to me that the extensions-required implementation is the preference of both a majority of the modules group and of the Node-using public. I think what would make me feel better about this conclusion, and would hopefully help the automatic-resolution proponents better accept it, would be more concrete indicators from our stakeholders (package authors, tool authors, general developers, etc.) that they agreed. The survey that @SMotaal tried to put together had the potential to give us this broader data, but that effort fell apart due to infighting within the group over what questions the survey should contain (because even deciding what to ask the public influences what answers you'll get). Ultimately this speaks to me of the dysfunction of this group: we can't even agree on a way to settle our disagreements.

So I guess where do we go from here? I think if the automatic-extension proponents are still interested in pursuing this, we can try to find consensus on a way to settle this question definitively, such as through some kind of democratic process like a survey of users or a vote of the modules group or of all Node contributors or all Node groups. Speaking for myself, that's what I would need to convince me that I'm wrong and that this should be changed. Likewise, if such a vote or survey reaffirms the current behavior, I would expect the automatic side to accept the election results and move on.

And it's not for nothing that the situation today is much better than it was when this issue was opened. Not only does "exports" allow pretty specifiers, but loaders allow opt-in to the automatic behavior even without the --es-module-specifier-resolution flag. All we're really arguing over at this point is whether the automatic behavior should be enabled by default rather than opt-in by users.

@weswigham
Copy link
Contributor

So, if you want feedback, I still get weekly pings asking me to modify TS to paper over the lack of extension searching, and to "automatically append" .js to the end of extensionless imports. It's anecdotal, sure, but I'm met with very hostile responses every time I say "just include the .js in the import yourself, it works fine". There's a massive collection of tools growing to add these extensions at build time, because people just don't want to start writing them. Of course we never got any kind of feedback here or anywhere else, this is an issue tracker for a working group, not a feedback forum or survey. But the TS issue tracker is our feedback forum (for better or worse), and I've been getting the brunt of this for months.

@guybedford
Copy link
Contributor

I've been following that specific issue for a while, and I also expected import './file.ts' in a src/app.ts folder to become a import './file.js' in a dist folder since a compiler by its nature is a transform of the source, which can surely include the file extensions. Having ./file.js in a TypeScript file to refer to file.ts seems odd to me and many users, and is certainly unintuitive. But I wouldn't push the cause of that problem entirely on Node.js by any means, since it's very much a TS tooling specific problem in being unable to support outputExtension or an output extension mapping system. It's also a documentation / awareness issue in TypeScript by taking such an unintuitive stance without very widespread education on it.

@devsnek
Copy link
Member

devsnek commented Aug 4, 2020

given that this is not a correctness issue, I think it is somewhat inappropriate to take a prescriptive approach to this. if you have some moral objection to code written for nodejs not working in browser when served with a static web server then you should convince people of that, not force it on them.

@weswigham
Copy link
Contributor

I've been following that specific issue for a while, and I also expected import './file.ts' in a src/app.ts folder to become a import './file.js' in a dist folder since a compiler by its nature is a transform of the source, which can surely include the file extensions.

As I've explained countless times, that's a leaky transform, which is why we won't do it. We do not, generally, override the runtime resolver (like webpack or parcel may do); what you see as your specifiers, is what you get in your output, and we're smart enough to know that a reference to ./a.js is a reference to the built output of ./a.ts. Now, we never had this question when people were able to elide extensions, since ./a could mean either, and people were OK with that, since it was the preferred style. Moreover, since we can't realistically append .js (or substitute .ts with .js) without bundling a runtime hook into your app that intercepts require (and potentially mucks it up for deno or ts-node), we couldn't find all such imports even if we tried, since imports have a dynamic component! So yes, there are ways to bypass the issue by complicating your runtime, the fact stands that people have an expectation for how the runtime should already behave.

We're not even being asked to support the "new way" of resolving esm specifiers, because we already do, because it's a subset of the old way (sans exports)! People are asking us to be able to write ./a in their source code, and have it mean ./a.js at runtime, like they want!

@guybedford
Copy link
Contributor

guybedford commented Aug 5, 2020

@weswigham I fully understand the reasoning and I'm not by any means wanting to inform TypeScript what to do. I'm simply stating my opinion that seems to be shared by others that this behaviour could be seen as unintuitive and there are possible alternative designs that could be seen to be more intuitive. For example, treating relative specifiers only (as defined in HTML) as permitting extension mapping through configuration could be one mechanism. The reason I'm stating this opinion is only to counter your assertion that Node.js put TypeScript in this situation, when we must look at the fact that TypeScript is making design decisions here, and the blame cannot be directed to Node.js, when making those design decisions without adequate user education is the underlying reason for confusion.

@weswigham
Copy link
Contributor

We are refraining from designing something to cover up what the runtime does not do, but users still want. That is very different than designing something in conflict with the runtime. The runtime doesn't support any kind of extension mapping thing, nor does it have a great need to. We do not want to be extending the resolver at compile time.

@guybedford
Copy link
Contributor

We do not want to be extending the resolver at compile time.

I must still admit I don't understand why. Also, isn't paths already an extension? TypeScript also has to integrate with the Node resolver for type lookups - is that not a resolver extension too?

@guybedford
Copy link
Contributor

Also, in theory a file mapping is not a resolver extension it is a file system remapping.

@devsnek
Copy link
Member

devsnek commented Aug 5, 2020

Let it be known that people who do not use typescript also dislike the lack of extension resolution.

@weswigham
Copy link
Contributor

I must still admit I don't understand why. Also, isn't paths already an extension?

  1. It only provides a way to look up type information - it provides .d.ts locations for .js files, since they (used to) often be held in a folder separate from normal dependencies; moreover this has no effects on paths actually in your code, or on runtime behavior.
  2. It was designed to be used with amd modules, not cjs or esm; it's from a time where bower was still in active use. Some people use it nowadays for local monorepo development, because the local package linkages there can be very nonstandard, and the monorepo layout/tool in use may not know how to handle TS types on its own. (Though normally just building them into your packages just works)

TypeScript also has to integrate with the Node resolver for type lookups - is that not a resolver extension too?

No, we reimplement the resolver wholesale; we run in browser runtimes where node's resolver isn't available, but we'll still analyze code intended for node. That's a small part of why we care so much, we have to maintain a parallel implementation of the thing, and not just the current version of it, but most recent past versions, too (and then provide flags for old/new behavior, depending - that's why we've been trying to wait for all the new stuff exports to stabilize, so it can all be behind one setting; so you could consider that support "extensions", but it'd feel disingenuous to say caring more about back compat then node itself qualifies as such). The more differences there are between past and current versions, the more confused our users get.

@guybedford
Copy link
Contributor

Putting these two arguments aside, the main argument to consider is that there is a file system mapping happening. The module at file:///path/to/project/src/file.ts is being mapped into a module at file:///path/to/project/dist/file.js (or file.mjs). Because the extension of the file is changing, the linkage between the modules before and after the transform is changing. The invariant that holds in all module systems is that relative paths with a file extension are always supported. Thus a file with a .ts extension can be remapped to a file with a .js or .mjs extension as part of the file remapping process so long as it is a relative specifier. All of this happens without any resolver extension being necessary and is the same type of mapping performed by standard build tools like RollupJS and esbuild code splitting.

@weswigham
Copy link
Contributor

weswigham commented Aug 5, 2020

All of this happens without any resolver extension being necessary and is the same type of mapping performed by standard build tools like RollupJS and esbuild code splitting.

Hold it there; those tools bundle their own resolver into the bundle. They no longer care about extensions at all - webpack even goes so far as just just use IDs to refer to each module anywhere it's precalculated the links. Point is, we don't bundle a big runtime like that.

@guybedford
Copy link
Contributor

guybedford commented Aug 5, 2020

I specifically did not mention Webpack - I'm referring to code splitting outputs from RollupJS and esbuild, which only rely on relative specifiers to exact file extensions as the basic primitive (with externals of course).

@gengjiawen
Copy link
Member

--es-module-specifier-resolution flag is unlikely works. Is it okay we put another field in package.json for this ?

@chyzwar
Copy link

chyzwar commented Aug 5, 2020

It seems to me that the real failure here wasn't in our inability to reach a consensus on this issue, as it's clear that there are diverging opinions that simply won't be reconciled; our failure was in not coming up with a definitive way to resolve this question. We created the --es-module-specifier-resolution flag with the expectation that user feedback would be clear enough and plentiful enough to tell us which way to go; but I think it's safe to say that that hasn't happened.

Because it is safe to say that nobody use native ESM in node. None of my projects at work or private can be converted. Most of the tooling lack proper support: berry, typescript, webpack, eslint, jest. Support is either partial or would require massive amount of work with little to no benefit. Lots of people have transpillers in theirs toolchain that compile ESM to CommonJS and are by large unaware. That why you do not see a lot of feedback. It is more visible in corresponding tools repositories where people are harassing maintainers.

I think many people like me just wait for things to "Just work". I am observing ESM progress in node for the last 3 years now from initial work in node-eps. This is moving incredibly slowly because of changes in resolution. You cannot expect adoption if a an migration is to change every relative import statement and every index.js. That why sooner or later all tools will be forced to support re-writing paths.

node.js was so late in the ESM party but want to change everybody assumptions about resolution.

@GeoffreyBooth
Copy link
Member

GeoffreyBooth commented Aug 5, 2020

I'm sorry, but I don't find any of these arguments persuasive; nor do I find this ongoing debate a productive use of our time. @ljharb at least opened this issue to try to collect feedback from across the web although it's been hijacked; and @rauschma opened a Twitter poll:

image

Excluding the “just show me the results” folks, that’s 39% for mandatory extensions, 30% for automatic resolution, and 31% that don’t care; from a sample of 294 votes.

I think efforts like this poll are the way forward to try to build support for any potential change, if one is desired.

@devsnek
Copy link
Member

devsnek commented Aug 5, 2020

people who prefer providing extensions can provide them regardless of what the default here is. what i see from that poll is 30% of devs saying they miss a functionality, that's pretty huge.

@weswigham
Copy link
Contributor

Seriously; if 30% of your users said they wanted something that didn't affect what the rest of your users could do, I don't know why you wouldn't consider it...

@chyzwar
Copy link

chyzwar commented Aug 5, 2020

Excluding the “just show me the results” folks, that’s 39% for mandatory extensions, 30% for automatic resolution, and 31% that don’t care; from a sample of 294 votes.

Again, another poll that is just misleading. Workaround only applies to external dependencies. Not only you suggested workaround is bad because lead to fragmentation but do not address core issue.

As result your node.js application written in typescript can end up with something like this

import module form 'main-module-package'; // with single main
import cjsSubmodule from 'main-module-cjs/submodule' //cjs modules still search for extension ?
import submoduleMapping from 'es-module-package/submodule'; // with subpath mapping
import submoduleNoMapping from 'es-module-package/submodule.js'; // without subpath mapping 

import base form '../base/index.js' // index.js as no longer supported
import local from './local-module.js' // relative imports need to have extension

Now in my typescript code I need to know how modules can be imported. For relative imports I need to import using transpilled extension and for existing code I need to refactor or import index.js.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
cjs features surveys Relates to things where people to you what you don't want to but need to hear
Projects
None yet
Development

No branches or pull requests