Support synchronous import.meta.resolve(…)

[lgarron]

🙋 feature request

Support import.meta.resolve(…).

🤔 Expected Behavior

Here is a code snippet with a set of examples that work in all modern browsers as of today. The argument to import.meta.resolve(…) in each case is a file in the source graph:

// JS
const libPath = import.meta.resolve("./lib/dynamic.js");
const lib = await import(libPath);

// JSON
const dataPath = import.meta.resolve("./data.json");
const data = await (await fetch(dataPath)).json();
console.log(data);

// CSS
const cssPath = import.meta.resolve("./themes/solarized.css");
const link = document.body.appendChild(document.createElement("link"));
link.rel = "stylesheet";
link.href = cssPath;

// Image
const imagePath = import.meta.resolve("./icons/errorsaurus.svg");
document.body.appendChild(document.createElement("img")).src = imagePath;

// Web worker
const workerPath = import.meta.resolve("./client/worker.js");
const worker = new Worker(workerPath, { type: "module" });
worker.postMessage("hi");

// WASM
const wasmPath = import.meta.resolve("./code.wasm");
const wasmModule = WebAssembly.compileStreaming(fetch(wasmPath), /* imports */);
console.log(wasmModule.exports.foo());

// Arbitrary resources
const resourcePath = import.meta.resolve("./resource");
const response = await fetch(resourcePath);
// binary
console.log(await response.arrayBuffer());
// text
console.log(await response.text());

In this case, it's valuable to:

1. Include the file in the source graph, ideally with zero config. (Particularly for .js and .wasm, the two web-native languages.)
2. In the case of a .js (or .ts, .css, potentially more depending on plug-ins) argument, treat it as an entry point and bundle as appropriate. For other formats, output the file in the app build, with the argument import.meta.resolve(…) appropriately re-written.

😯 Current Behavior

import.meta.resolve(…) is not recognized and handled.

💁 Possible Solution

I know Parcel uses swc, but it's not clear to me how much of the appropriate source graph functionality is implemented in swc or Parcel. I've also filed an issue at swc-project/swc#7155

🔦 Context

This functionality is critical for library authors (like me) who want to use web workers and WASM to ensure that end users have a good experience. As a library author, I want to ensure my code works in Parcel, but I don't have direct control over the Parcel configuration that apps use. So it's important to have something that works out of the box, and import.meta.url(…) is an unambiguous way to specify relative resources.

This would also help resolve #7623 , in that it moves away from per-bundler hacks to a single pattern that is unambiguous and well supported in browsers.

💻 Examples

See above.

[lgarron]

For a set of test cases, see:

git clone https://github.com/lgarron/loadeverything.net && cd loadeverything.net
git checkout --detach d5dfd056f81c8c0c020f2962741fd325ed4c67ea

make test-parcel

[mischnic]

So it should behave like the existing new URL("npm:./data.json", import.meta.url)?
And yes, this has to be done in Parcel and not swc.

[lgarron]

So it should behave like the existing new URL("npm:./data.json", import.meta.url)?

Yeah, it's fairly similar. For relative paths (starting with ./ or ../) like the following, code like the following prints the same result twice:

console.log(import.meta.resolve("./lib/dynamic.js"));
console.log(new URL("./lib/dynamic.js", import.meta.url).href);

import.meta.resolve(…) also supports bare specifiers like import.meta.resolve("comlink"), although I don't need that functionality for Parcel to be compatible with my libraries.

[devongovett]

Feels like there are two possible meanings for import.meta.resolve, and I'm not sure which is the right one.

1. As you wrote here, create a dependency on an isolated bundle and return a URL to it, exactly the same way as new URL.
2. Return the id of the resolved module, like require.resolve. Or maybe the URL of the bundle the module is in, plus the module id somehow (e.g. hash or query param).

The first doesn't seem particularly useful given that new URL already exists. It also probably wouldn't do what you want in your examples. For one, whenever we return a URL, we don't know how that URL will be used. You could inject it into a <script> tag, load a Worker, an <iframe> or literally anything else. Therefore, we have to mark the bundle it creates as "isolated". This means it cannot share any dependencies with other bundles (because it might be loaded in a different environment without access to them), resulting in duplicate code being loaded. We also don't know what environment it will be loaded in, meaning we might use incorrect APIs to load other bundles (e.g. DOM APIs in workers or visa versa). new URL also has the same problems. That's why new Worker(new URL(...)) works correctly, but new Worker(someURLVariable) doesn't. Another problem is that this would mean that import.meta.resolve would have an effect on code splitting (similar to dynamic import()). Rather than simply returning what the resolution is, it would also change the resolution.

The second seems a bit more useful, and more in line with the intent of the function. It would only return what the resolution is, without affecting code splitting, and matches its CJS counterpart require.resolve. This could let you create "weak" dependencies, where if you include modules elsewhere in the build you can resolve them at runtime, but if not they don't get bundled.

Seems like Webpack has the same questions: webpack/webpack#16693. Perhaps it would be good to get alignment between bundlers on this before implementing it one way or the other.

[lgarron]

Therefore, we have to mark the bundle it creates as "isolated". This means it cannot share any dependencies with other bundles (because it might be loaded in a different environment without access to them), resulting in duplicate code being loaded.

Hmm, I'd really love to see any approach that allows import.meta.resolve(…) to be treated more like an import for bundling, particularly if the argument is a .js or .wasm file, i.e. essentially option 1.

To me, the value of import.meta.resolve(…) (with new URL(…, import.meta.url).href as a fallback) is that it allows authors to create a declarative source graph that works across all environments. It is critical for both performance and correctness that module workers can share their dependencies with non-worker code.

For example, I measured that a page like https://experiments.cubing.net/cubing.js/mark3/ needs three times the code if module splitting does not work across worker and non-worker code, which is a significant impact on end users, especially on cell networks or other slow connections.

If I'm publishing a such an app myself, I can choose a bundler with settings that allows this to work. But it's very important to me that this also works out of the box for anyone using my underlying library in their own app. I'd like to be able to recommend Parcel as a zero-config bundler, and this functionality is table-stakes because there is no portable alternative syntax.

The first doesn't seem particularly useful given that new URL already exists. It also probably wouldn't do what you want in your examples. For one, whenever we return a URL, we don't know how that URL will be used.

This was a concern in #7623 , which has stalled. Given that import.meta.resolve(…) is clear in name and semantics I'd like to advocate that import.meta.resolve(…) is implemented in a way that actually does resolve the import URL to its runtime location.

That's why new Worker(new URL(...)) works correctly, but new Worker(someURLVariable) doesn't. Another problem is that this would mean that import.meta.resolve would have an effect on code splitting (similar to dynamic import()). Rather than simply returning what the resolution is, it would also change the resolution.

I believe by far the best option is for bundlers to change the resolution and be involved in code splitting, so that the runtime resolution still points to the appropriate bundled entry file/resource. As far as bundling is concerned, it's essentially like a dynamic import (that happens not to actually load the import directly).

Seems like Webpack has the same questions: webpack/webpack#16693. Perhaps it would be good to get alignment between bundlers on this before implementing it one way or the other.

Thanks, I'll try to advocate for the same over there. Library authors like me will not be able to publish performant & portable ESM worker and WASM code unless bundlers support this consistently.

[lgarron]

If my reasoning is not persuasive, could I ask if you can think of an alternative way to publish code like the following to npm, so that it works directly in node, browsers¹, and bundlers?

// Web worker
const workerPath = await import.meta.resolve("./client/worker.js");
const worker = new Worker(workerPath, { type: "module" });
worker.postMessage("hi");

// WASM
const wasmPath = await import.meta.resolve("./code.wasm");
const wasmModule = WebAssembly.compileStreaming(fetch(wasmPath), /* imports */);
console.log(wasmModule.exports.foo());

Note that:

• import wasm_module from "./code.wasm"; does not work in node or browsers, so it is not an option. (I'd love to see a proposal that does this with import assertions. But that likely wouldn't land in major browsers for at least a year, whereas the snippet above works in all browsers today.)
• The code that instantiates a WASM module is often wrapped in fallback alternatives (e.g. a fetch of wasmPath falling back to node's readFile), and is auto-generated by WASM toolchains in a lot of cases. It's not as practical to rely on a single inline instantiation pattern like a new URL(…, import.meta.url) call directly inside a specific surrounding function.

Footnotes

1. Although browser code from npm is often passed through bundlers, there are CDNs that allow you to import directly. In general, it's also valuable in general if the published build works directly in browsers, for more consistent maintenance and debugging. ↩

[devongovett]

new Worker(new URL('./client/worker.js', import.meta.url), {type: 'module'})) should work today everywhere I think.

Hmm, I'd really love to see any approach that allows import.meta.resolve(…) to be treated more like an import for bundling

We could have it work like import() but this would not work for workers. Normal dynamic imports run in the same JS environment as the parent module, and therefore have access to the modules that have already been loaded on the page. Workers run in a separate module environment, so there will be different instances of the same modules. This requires bundling to work differently as well. We can't reuse any modules loaded outside the worker (from a parent bundle), we would need to load them again inside the worker. That's what we do already with new URL.

Given that import.meta.resolve(…) is clear in name and semantics

I think it's clear what this should do in an unbundled environment, but unclear what bundlers should do with it. Once you start having multiple modules in the same file, it gets murky what a "URL to a module" means.

[lgarron]

new Worker(new URL('./client/worker.js', import.meta.url), {type: 'module'})) should work today everywhere I think.

I really wish it did, but unfortunately it very much does not. 😭

My impression is that bundlers have some reservations about supporting this syntax, particularly when the argument is not a relative path. I think import.meta.resolve(…) addresses those.

In any case, is there anything you consider a cross-compatible approach for WASM?

We could have it work like import() but this would not work for workers.

Hmm, I think theres a misunderstanding here? import.meta.resolve(…) very much does not perform an actual import. By "treated more like an import for bundling" I mean "treated like an import for the purpose of analyzing and modifying the module graph".

I think it's clear what this should do in an unbundled environment, but unclear what bundlers should do with it. Once you start having multiple modules in the same file, it gets murky what a "URL to a module" means.

Suppose you turn every import.meta.resolve(…) argument from the source code into a separate entry file in the output graph. Is there any murkiness left at that point?

[devongovett]

treated like an import for the purpose of analyzing and modifying the module graph

Yep that's what I meant. Say you have an input graph like this:

// index.js
import {value} from './value';
const asyncValue = (await import('./async')).value;

value === asyncValue

// async.js
export {value} from './value';

// value.js
export const value = {};

In this case, the value and asyncValue will be equal because value.js will point to the exact same instance of the module. When we bundle a graph like this, you'd get an output like this:

• index.bundle.js, containing index.js and value.js
• async.bundle.js containing only async.js

async.bundle.js can assume that index.bundle.js is already loaded when it loads, and therefore the value.js module will already be loaded. This means value.js doesn't need to be split out into its own bundle.

If async.js were instead a worker, it could not access an existing module instance for value.js, so then you'd have to bundle like this:

• index.bundle.js, containing index.js and value.js
• async.bundle.js containing async.js and value.js again

or like this:

• index.bundle.js, containing only index.js
• async.bundle.js, containing only async.js
• value.bundle.js, containing only value.js (shared between the two above)

My point was simply that bundling import.meta.resolve as if it were import() would lead to the first result above, where it would assume that value.js is available in a parent bundle that's already loaded. If we want it to work in workers, we'd need to do one of the other options below, which is different, and matches how new URL works today.

[devongovett]

I really wish it did, but unfortunately it very much does not. 😭

I'm curious about these tests. For example, I'm pretty sure new URL should work with JS, JSON, and WASM in Parcel without any config. What issues did you run into?

[lgarron]

If async.js were instead a worker, it could not access an existing module instance for value.js, so then you'd have to bundle like this:

Hmm, I don't entirely follow what this means. But in any case, I agree that it would be a surprise if value === asyncValue didn't hold (both outside a worker and inside it). If Parcel calls this "new URL" semantics, sounds fine to me. 😝

I really wish it did, but unfortunately it very much does not. 😭

I'm curious about these tests. For example, I'm pretty sure new URL should work with JS, JSON, and WASM in Parcel without any config. What issues did you run into?

I'm not entirely sure, as I now seem to have issues with the Parcel build loading any code at all?

git clone https://github.com/lgarron/loadeverything.net && cd loadeverything.net
git checkout --detach d5dfd056f81c8c0c020f2962741fd325ed4c67ea

make test-parcel

This runs npx parcel build --public-url . --dist-dir "dist/test-parcel" test/index.html test/verbose/index.html as part of make test-parcel, i.e. a pretty bog-standard invocation.

It results in an empty console. Not even "Pause on caught exceptions" catches anything, and I can't figure out why. 😳

[lgarron]

If we want it to work in workers, we'd need to do one of the other options below, which is different, and matches how new URL works today.

@devongovett, would you support a PR to implement this functionality?

It would be really valuable for Parcel to support this, so I'd be willing to spend quite some time on it.

[lgarron]

@devongovett: I looked around the Parcel codebase for about an hour last night to see if I could put together a PR, but I couldn't figure out how to mirror the new URL(…, import.meta.url) behaviour for import.meta.resolve(…). Could I ask for some pointers of which parts of the code would be best to try to do this in?