Proposal: import.meta.resolve (was: import.meta.resolveURL)

[guybedford]

Motivation

With import.meta.url support in Module Scripts, common asset loading workflows look something like the following:

const response = await fetch(new URL('../hamsters.jpg', import.meta.url).href);
const worker = new Worker(new URL('./worker.js', import.meta.url).href, { type: 'module' });

With an import.meta.resolveURL function, these workflows can be simplified to:

const response = await fetch(import.meta.resolveURL('../hamsters.jpg'));
const worker = new Worker(import.meta.resolveURL('./worker.js'), { type: 'module' });

much more clearly indicating the intent of these common loading scenarios.

The added benefit of the above is that we bring static analyzability back to these workflows, in that build tools can now statically determine where assets and workers are being resolved and handle rewriting of these kinds of expressions. In this way assets and workers can much more easily be updated to point to optimized resources during builds than requiring build tools to try and analyze custom URL manipulations, which may vary more than the initial examples above and lead to more unreliable results. This would, for example, be useful for us in Rollup to be able to provide these optimizations more easily to users.

Proposal

The proposal is a function of the following form:

resolveURL (url: string): string

Which returns the WhatWG URL normalization applied for url relative to import.meta.url, and returning the fully normalized URL string.

I'd be happy to assist with any spec work here as well.

Feedback welcome!

[annevk]

Bikeshed: I'm not sure if we want to continue to use "resolve" for this, since the URL Standard doesn't really have that concept. The alternative would be "parseURL".

[domenic]

I'm strongly in favor of this, although I realize there is indeed a bikeshedding question here. /cc @whatwg/modules for thoughts, especially implementer interest.

Spec-wise, it seems like you'd create one function object per module script... no other real choices I can see. Seems fine. Might be a bit fun figuring out whether to use any IDL formalisms or just JS style spec.

The bikeshedding is indeed probably the biggest issue here. I think "resolve" is the typical user-exposed name for this operation in a module context; see e.g. Node.js or Webpack. (I thought this went back to the CommonJS standard or AMD as well, but I can't find any evidence.) I guess that's more about module identifiers though.

I think we should just run with that, in fact: we should just use the name "resolve", and let this it work on all module resolution. This doesn't matter now, but with package name maps it would then allow nice things like import.meta.resolve("lodash/fp.js").

[dcleao]

@domenic Will there be a way to know the package name of the current module/package, something like import.meta.packageName?

[hiroshige-g]

import.meta.resolve("lodash/fp.js")

This looks nice, but conflicts with relative path URLs without leading .. For example,

new URL('hamsters.jpg', import.meta.url).href

returns something like https://example.com/baseurl/hamsters.jpg while

import.meta.resolve('hamsters.jpg')

might tries to resolve hamsters.jpg as a package name.

Two functions might be needed (one for parseURL and one for #resolve-a-module-specifier).

[hiroshige-g]

@dcleao I expect No.
A module script can be imported using multiple names or a URL from multiple module scripts, and thus we can't reliably/deterministically determine which name is to be used.
There are similar concerns related to import.meta.scriptElement https://github.com/tc39/proposal-import-meta/blob/master/HTML%20Integration.md.

For example,

• Both packageName1 and packageName2 are resolved to https://example.com/foo.js by a package name map.
• a.js, b.js and c.js imports packageName1, packageName2, and https://example.com/foo.js, respectively, and all of them result in a single foo.js to be loaded.

In this case, it's not clear what import.meta.packageName should return in foo.js (packageName1, packageName2, or null?).

[benjamingr]

I've also been following this discussion on the nodejs modules side and am strongly in favour of this. This would simplify a lot of workflows for people using our new APIs (like Worker, mentioned above). I think it could also bring similar benefits to people writing code on the client.

[domenic]

@hiroshige-g you're right about the potential "conflict". However, I am not sure there are any consumers who really insist on typing import.meta.resolve('hamsters.jpg') instead of import.meta.resolve('./hamsters.jpg'). In contrast, giving web authors two methods which behave very similarly, with slight differences around how they treat unprefixed strings, seems like a high burden.

[bmeck]

I have concerns with this being Synchronous if there is intent for this to be shared with Node in any way. The current design of ESM loaders in Node is with asynchronous resolution. If this API was asynchronous it seems like it wouldn't introduce a bottleneck if a microtask was added to resolve when compared to the cost of fetching/loading a Module. That said, I think this API should return a Promise and be asynchronous.

Per import.meta.resolve('hamsters.jpg'), I would be surprised if this worked currently because hanging the API off of import. I would expect it to work like import resolution does; just as @domenic says I think there is a burden we can avoid by making these things match.

In addition, without it being more than just sugar for new URL(..., import.meta.url) it seems like this convenience API isn't really worth it. Whenever/however bare specifiers are supported and this API no longer is such a simple sugar it seems more valuable.

[domenic]

Yeah, we wouldn't make this async on the web, so Node may not be able to match, it's true. Node may want a different name for its async resolution algorithm.

[bmeck]

@domenic I'd prefer to use resolve() our caching and import.meta.url already don't 100% match so I think divergence already occurred. I'm not sure why it wouldn't be ok as async on the web, if you could explain that.

[domenic]

On the web URL and module specifier resolution is synchronous.

[bmeck]

@domenic that doesn't really explain why matching would be problematic? Also, is that always going to be the case? Wouldn't there be possible long lookups if loading a package name map, or having an overly large package name map?

[guybedford]

I don't agree that module specifier resolution should be synchronous on the web, as I don't think package name maps should block execution, but rather utilize an asynchronous resolver to avoid this.

[domenic]

Matching would be problematic for the same reason making new URL async would be problematic.

And yes, on the web it would be a bad performance degradation to make resolution async, so new things like package name maps will not be introduced to do that.

[guybedford]

I'd suggest we wait on this then until there is a clear solution that doesn't create divergence or performance degradation.

[domenic]

As @bmeck points out, there is already divergence, so I don't think we should wait on that. We can avoid performance degradation easily by making it sync. So I think the solution is pretty clear.

[bmeck]

@domenic

Matching would be problematic for the same reason making new URL async would be problematic.

I don't understand this comment. new URL is a "frozen" resolution mechanism, whereas module resolution seems to be actively growing and becoming more complex over time.

And yes, on the web it would be a bad performance degradation to make resolution async, so new things like package name maps will not be introduced to do that.

Resolution could stay sync under the hood, I'm just saying exposing the results to JS userland as Promises would allow unification rather than divergence.

In addition as @guybedford was pointing out in that alternate thread, blocking Modules seems like degradation already by forcing it to be sync...

[bmeck]

@domenic

We can avoid performance degradation easily by making it sync. So I think the solution is pretty clear.

I don't see how making it sync increases performance if package name maps are forced to be prevent Module loading. Perhaps you have a writeup? Currently, it seems like the inverse of what you are stating would be more beneficial.

[domenic]

Yeah, as I mentioned in WICG/import-maps#48, we are working on porting the writeup/explorations we did in the Blink prototype to the public space. Stay tuned there for updates.

We don't want to expose things that are synchronously available as promises. If someone needs to treat a sync result as a promise for e.g. Node.js compatibility, they can use Promise.resolve to do that.

[matthewp]

Forcing async also makes some use cases difficult (or at least less ergonomic) without top-level await. I would probably just use new URL if resolve() were async.

[dcleao]

Thanks @hiroshige-g.
About import.meta.packageName, I was making a parallel with RequireJS, where we have module.id. It's sometimes useful to know "your" module id, so that you can determine the module id of, for example, sibling modules, using a relative module id. In RequireJS, this cannot be accomplished using the url, cause the actual url depends on whether bundling is used or not.
These issues might not be a problem here.

[benjamn]

Is there any chance of allowing import.meta.resolveURL(id, parentURL) to specify a parentURL that's different from import.meta.url?

I'd like to be able to follow a chain of relative imports in a reliable way (respecting import maps and all that), and it doesn't seem possible to retrieve the import.meta.resolve function from an arbitrary module.

I'm happy to be redirected to any prior discussion of this topic, of course.

[domenic]

Is there any chance of allowing import.meta.resolveURL(id, parentURL) to specify a parentURL that's different from import.meta.url?

For import.meta.resolveURL (the topic of this thread), what you propose would be identical to new URL(url, parentURL).

For import.meta.resolve (a different thread), it would not, since the first argument could be a specifier, not just a URL. But for resolveURL there would be no difference.

[benjamn]

@domenic Ahh, thanks for clarifying that distinction.

Are you envisioning a world where we have both import.meta.resolveURL and import.meta.resolve at the same time (at least in browsers)? I don't have any objections to having both, I'm just curious about the relationship between the proposals, if you have thoughts.

By the way, @guybedford directed me to his PR adding support for import.meta.resolve to Node.js, and it looks like my use case will be covered by that API (at least for Node): WICG/import-maps#79 (comment)

[domenic]

if there are strong motivations to do so (that's for ergonomics, right?)

Ergonomics, simplicity, and perhaps a tiny bit of performance. In particular it's not ideal if using import.meta.resolve means that you and your entire call stack have to become async functions. (See https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ for a long-winded explanation of this.)

Interaction with pending external import maps: I was also thinking of this, but I've just noticed that we don't have to care about this: [...] Therefore, import.meta.resolve() don't have to wait any pending import maps. Is this correct?

Agreed; that was my understanding as well. The only thing I can think of is if we wanted to support adding new import maps after the initial ones load, but that is fraught with danger (as it changes how the same specifier resolves over time).

[benjamingr]

Hey, regarding sync vs. async import.meta.resolve, I'm a bit confused because I would have expected browsers to want to push for an async API and Node for a sync one:

• Node can typically always block on things it can resolve quickly (like it does on require) - can't it? So I would expect Node to always be able to work around a sync import.meta.resolve.
• Browsers typically can't block on I/O (especially since it's network I/O) so anything that would require I/O for import.meta.resolve would be prohibited.

This seems to be the case for:

• Following redirects in URLs.
• Do HTTP -> HTTPs rewrites under certain policies.
• Wait on loading further import maps and other extensions to the mechanisms browsers might have in the future
• Contain any logic that requires capabilities related to the current page policy.

So mostly - I'm confused :]

I personally kind of hate async APIs but since this API doesn't take a function and if it throws will always be the last developer sees in their stack trace - I'm not too concerned. Any time I'd use this I'm already likely before an import or in an async function.

(I have absolutely no opinion about the name and my opinion about whether should resolve be sync is obviously not made up + doesn't really matter here as I hold no position in whatwg or Node's (active) modules team)

[domenic]

Following redirects in URLs.

Resolution of module specifiers does not follow redirects.

Do HTTP -> HTTPs rewrites under certain policies.

Resolution of module specifiers does not take into account network-layer concerns like HSTS.

Wait on loading further import maps and other extensions to the mechanisms browsers might have in the future

As discussed above, we think this is unlikely, because it would mean that a module specifier could change its resolution over the lifetime of the program. All import maps need to be present before any module scripts start executing (and thus, before any module resolution happens).

Contain any logic that requires capabilities related to the current page policy.

Resolution of module specifiers doesn't require any information about the current page's capabilities. It only requires the referring module script's base URL (currently) and its import map (in the future).

[benjamingr]

@domenic thank you for clarifying I think I understood what resolve does here incorrectly then. I assumed it would be like Node's require.resolve in that it would "resolve" the URL I would eventually get if I importd a file.

In retrospect I'm not sure why I assumed that since the initial post and proposal just says "returns the WhatWG URL normalization applied for url relative to import.meta.url, and returning the fully normalized URL string.".

Now I'm just confused about why Node would want this asynchronously if import.meta.url is known.

[domenic]

Well, it is a bit confusing. The original post, and some of the discussion here, seems to be discussing sugar for (new URL(x, import.meta.url)).href. But import.meta.resolve(), as proposed (e.g. concretely in #5572) does the full "resolve a module specifier" algorithm. This does not include network-level concerns like redirects or HSTS, but it is also not just sugar for URL resolution, as it includes things like prohibiting bare specifiers (now) or import maps (in the future).

[guybedford]

I am quite worried about a sync import.meta.resolve inhibiting performance optimizations in future where we might want to dynamically load import maps.

Consider the case of a large application where the import map is very large. The import map blocking all scripts from executing on the page then becomes the critical performance bottleneck and concern for the entire application performance.

In these applications deferring parts of the import map until they are needed is an incredibly useful optimization, and this is enabled by allowing import maps to be deferred after initial bootstrap executions.

When the import map is deferred, the problem is how to handle resolution when there are "inflight" import maps. It seems like there are a few options here:

1. import.meta.resolve is always sync, and will resolve whatever import map has already loaded right now. If an import map is in-flight with new resolutions, those resolutions will fail to resolve until those dynamic import maps have fully completed loading at which point the sync resolve would respect them.
2. import.meta.resolve is async, and will always wait for any "inflight" import maps to complete or error before completing the resolution process. This still leaves some timing gap where a new import map loaded dynamically after the resolve call may be missed.
3. import.meta.resolve is async, and will hook out to a global callback to ensure resolutions are ready. This would actually be my preferred solution - document.registerResolveCallback or something like that. It allows applications to control what resolutions are available and when and effectively enables in-browser package management for platforms to load non-predetermined code.

I'm not sure this is the best place to dive into these deep topics now, but I'm also not sure where that place is - since no one has yet created an open standards forum for discussion and consensus-building around these topics.

For now, for the reasons above I would be very weary shipping a synchronous resolver.

I've chatted briefly with @yoavweiss and @domfarolino about these optimization concepts before, and I hope we can make a decision here that doesn't restrict performance optimization of the web in future.

[domenic]

I'm not really comfortable with any form of deferred import maps, since that breaks the idempotency guarantees of module specifier resolution. Once modules have executed, the import maps need to be locked.

[guybedford]

@domenic this has nothing to do with what you are or are not comfortable with! This is about the web and what is best for future performance. If you want to drag this argument into the weeds of how to define an immutable import map extension process, I'd be glad to do that - but let's not ship anything based on "disallowing discussion" please.

[domenic]

I don't see any need to make this personal, based on my turn of phrase. Let me be clearer: I would be opposed to shipping such a thing in Chrome, or incorporating such a thing into the HTML Standard. I am confident it is possible to have performant web applications without changing the meaning of module specifiers over time.

I'll step back from this discussion as it seems like it's gotten unproductive.

[benjamingr]

@guybedford

I am quite worried about a sync import.meta.resolve inhibiting performance optimizations in future where we might want to dynamically load import maps.

I'm not sure I understand, so you'd expect .resolve to automatically "block" until all resource maps have downloaded?

Even assuming you could load import maps dynamically - wouldn't it be better for resolve to be sync in this case and for the user to explicitly wait for the loading operation?

What about a sync .resolve and if dynamic import maps ever get introduced, a document.registerResolveCallback (or a similar API) for knowing when import maps resolved so you can .resolve safely?

If I understand #3871 (comment) correctly .resolve (I probably don't) really shouldn't be doing much work.

[guybedford]

Thanks @benjamingr for engaging in the discussion on these questions.

I'm not sure I understand, so you'd expect .resolve to automatically "block" until all resource maps have downloaded?

Import maps, as specified today, will always block all module scripts from executing until the entire import map has completed loading. At that point, no further import maps can ever be loaded into the page.

Even assuming you could load import maps dynamically - wouldn't it be better for resolve to be sync in this case and for the user to explicitly wait for the loading operation?

It depends - imagine you have a core app and third-party user plugins. The core app loaded with a boot time import map. The third-party user plugins are currently being installed with an in-flight import map. Now if your core app wanted to determine where a third-party user plugin asset is located (the major use case of import.meta.resolve being able to determine asset paths), surely waiting for the import map to complete would be beneficial? It feels like asking a user of dynamic import() to first do load() before they can execute.

A sync resolve can certainly work with the above workflow with the extra userland wait step like with the API you suggest.

To be completely honest I'm not 100% what is best here - it helps a lot to discuss it!

@domenic I just think there's a lot of depth to the concerns here, and am frustrated yes that it isn't possible to openly discuss it anywhere. Suggestions welcome re the right forum.

[benjamingr]

Thanks I think I understand the difference in perspectives now.

Honestly from an ergonomics standpoint I would prefer .resolve to by sync but that's just my Node.js PoV and I haven't really written many apps that are like the ones you describe nor have I considered that use case in depth.

I definitely see a challenge in dependencies wanting to use import maps themselves with sync resolution.

So IIUC the use case we are concerned with is:

• Import maps exist.
• Import maps are dynamically loadable.
• I have an app using import maps and that app is loading a third party module which is also using import maps.
• I want to .resolve to get the URL of an asset mapped in the second import map.

Wouldn't I have a place where I would have to "import" said second map? How would the resolve call even know how to wait? (If it waits, what happens if I add a map while it's waiting? What if it resolves and then I load a module?)

I would expect an async resolve to get a dependency list or similarly be explicit about what it's waiting for. Otherwise even if it's async it would be very easy to .resolve before an import map is "in flight" wouldn't it?

(Keep in mind I am very likely the least knowledgeable participant on this topic here - and if you run out of patience feel free to disengage, I doubt I will be able to contribute much philosophy and don't have a lot of context, I am just asking a bunch of questions and learning a bunch)

[guybedford]

The model of dynamic import maps extension can be made well-defined by considering the following:

1. There is a large underlying import map for the application (no limits - it could be as large as the whole npm registry if you want). This map is well-defined with scopes and imports all uniquely set to their best versions, like any standard package lock file we use today with npm.
2. When loading a page of the application, you only load the "slice" of the import map necessary for the initial rendering of the page. This is a performance optimization.
3. Whenever loading a new bare specifier that would otherwise throw like import('newpkg') - you have a browser hook which gives the application loader a chance to load and inject that new slice of that larger immutable import map with the resolution information for newpkg. We are only adding new mappings which would otherwise throw and none of those mappings have been resolved yet by definition of the hook catching their resolutions first.

In this way you can progressively load an application import map and modules based on the user interactions with it.

The dynamic resolver hook here is key - the resolver is a very natural home for enabling this import map injection in a well-defined way - as we need to be able to have all module resolutions to the new slices be able to trigger and wait on this hook for the whole import graph, while we load the full dependency information.

If we make import.meta.resolve sync now, then it would always be the current slice image, and wouldn't actually expose the "true resolver" for the application. require.resolve in Node.js looks up things in node_modules. Imagine if it only checked require._cache based on what's already loaded. That's the difference in this model.

[benjamingr]

Ok, I've slept it over and I think I (generally) understand - thanks. That mental model (of the large immutable map we load chunks of vs. a mutable map) sounds pretty cool.

I honestly don't know if having a synchronous resolve (and in that model should it ever be implemented throw an ERR_RESOLUTION_MAP_NOT_LOADED or something) is better or worse than an asynchronous resolve.

If "dynamic" import maps are ever supported - I would expect import and resolve calls to be bound to a list of import maps (a specific "subset" of that large underlying map).

Otherwise in the above example if the import map is "as large as the whole npm registry" I would definitely need a way to "unload" parts of the import map or otherwise manage what parts are loaded.

I think I understand now why we'd want an async .resolve in that model (of it and dynamic import being points to wait for loading) but I am not sure about:

Wouldn't I have a place where I would have to "import" said second map? How would the resolve call even know how to wait? (If it waits, what happens if I add a map while it's waiting? What if it resolves and then I load a module?)

I would expect an async resolve to get a dependency list or similarly be explicit about what it's waiting for. Otherwise even if it's async it would be very easy to .resolve before an import map is "in flight" wouldn't it?

[guybedford]

If "dynamic" import maps are ever supported - I would expect import and resolve calls to be bound to a list of import maps (a specific "subset" of that large underlying map).

This brings up the critical point actually - it is actually crucial in a dynamic import maps model that import will wait on those deferred import maps before finishing resolution.

import and dynamic import() are exactly the lazy trigger points you want for fetching a new slice of the resolution maps. Otherwise web tooling is back to constructing its own custom wrappers around every dynamic import in userland code to ensure these optimizations (and note that graph preloading is also associated with import triggers).

This is done in the spec through the dynamic import and import host hooks, and having the resolver itself form the asynchronous blocking point for import map readiness, before resolutions are completed.

Thus, in the HTML spec, the resolver being async allows for optimized lazy loading with deferred import maps. And this is my concerns with exposing a sync resolver call, as that might exactly block this future work.

[benjamingr]

it is actually crucial in a dynamic import maps model that import will wait on those deferred import maps before finishing resolution.

If I don't explicitly have a dependency between an import and a module map then I don't understand how the import'/resolve` would know which slices to wait for since while the slice it could come from could be "in flight" it would not be the one triggering the fetch of that map (because it would not be aware of it, right?).

Or are you saying what import map each specifier comes from is established ahead of time (in some "parent import map" and the import knows where to fetch the map from if it "throws"?

[guybedford]

One way that import can know if a package is not in its existing map is that the concept of bare specifiers is already defined in HTML.

If a user attempts to load a bare specifier that does not have a map, it would throw an error. Having a global event / hook that can handle this case before the error can allow dynamic injection of just in time resolution information.

This does contrast with allowing arbitrary URL mappings in import maps though, a feature of import maps I am against actually, for related reasons.

@benjamingr I'm glad we could engage in some of these discussions here given that there haven't been other avenues to date. Instead of continuing this related tangent further, perhaps we can aim to bring more of these discussions over to the WICG discord in future if that seems the right home for this stuff.

[RReverser]

Nit: can we / should we rename this issue to match the (agreed upon?) import.meta.resolve naming? Would help for easier search across issues.