Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Batch of several breaking changes #2508

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Batch of several breaking changes #2508

wants to merge 2 commits into from

Conversation

evanw
Copy link
Owner

@evanw evanw commented Aug 31, 2022

This is a batch of several breaking changes. I'm planning to release them together in an upcoming release. TL;DR: new URL(...) and glob import(). In detail:

  • Interpret and rewrite new URL(..., import.meta.url) expressions when bundling (#312, #795, #2470)

    Some other bundlers have adopted a convention where the syntax new URL('./file.js', import.meta.url) causes file.js to be included in the current bundling operation as an additional entry point. The './file.js' string is rewritten in the bundler's output to point to the resulting generated file for that entry point in the output directory (relative to the generated file containing the new URL(...) syntax). This is somewhat similar to how import('./file.js') works except that this reference just returns a URL object without importing the module. That lets you pass the URL of a module to other APIs such as new Worker(...) that take a script URL as input.

    Previously this pattern didn't work at all with esbuild, but it will now work in esbuild starting with this release. To use it you must ensure that bundling is enabled, that the output format is set to esm, and that code splitting is enabled (so you need to use --bundle --format=esm --splitting). In addition, the path must be a relative path (i.e. it must start with either ./ or ../). Here's what using this feature looks like:

    const url = new URL('./worker.ts', import.meta.url)
    const worker = new Worker(url, { type: 'module' })
    
    worker.onmessage = (event: MessageEvent) => {
      console.log(event.data)
    }
  • Handle import paths containing wildcards (#56, #700, #875, #976, #2221, #2515)

    This release introduces wildcards in import paths in two places:

    • Entry points

      You can now pass a string containing glob-style wildcards such as ./src/*.ts as an entry point and esbuild will search the file system for files that match the pattern. This can be used to easily pass esbuild all files with a certain extension on the command line in a cross-platform way. Previously you had to rely on the shell to perform glob expansion, but that is obviously shell-dependent and didn't work at all on Windows. Note that to use this feature on the command line you will have to quote the pattern so it's passed verbatim to esbuild without any expansion by the shell. Here's an example:

      esbuild --minify "./src/*.ts" --outdir=out

      Specifically the * character will match any character except for the / character, and the /**/ character sequence will match a path separator followed by zero or more path elements. Other wildcard operators found in glob patterns such as ? and [...] are not supported.

    • Run-time import paths

      Import paths that are evaluated at run-time can now be bundled in certain limited situations. The import path expression must be a form of string concatenation and must start with either ./ or ../. Each non-string expression in the string concatenation chain becomes a wildcard. The * wildcard is chosen unless the previous character is a /, in which case the /**/* character sequence is used.

      // These two forms are equivalent
      const json1 = await import('./data/' + kind + '.json')
      const json2 = await import(`./data/${kind}.json`)

      This feature works with require(...), import(...), and new URL(..., import.meta.url) because these can all accept run-time expressions. It does not work with import and export statements because these cannot accept run-time expressions.

@lgarron
Copy link
Contributor

lgarron commented Aug 31, 2022

I'm really looking forward to this, thanks for the URL change in particular!

I've tried testing this locally by doing make build in esbuild repo and linking the binary into node_modules/esbuild-darwin-arm64/bin/esbuild and node_modules/esbuild/bin/esbuild for a project. Unfortunately, that fails for me:

/Users/lgarron/Code/git/github.com/cubing/cubing.js/node_modules/esbuild/lib/main.js:1624
  let error = new Error(`${text}${summary}`);
              ^

Error: Build failed with 3 errors:
error: panic: Internal error (while parsing "src/sites/experiments.cubing.net/cubing.js/twisty/simultaneous.ts")
error: panic: Internal error (while parsing "src/sites/experiments.cubing.net/cubing.js/play/index.ts")
error: panic: Internal error (while parsing "src/cubing/search/instantiator.ts")
at failureErrorWithLog (/Users/lgarron/Code/git/github.com/cubing/cubing.js/node_modules/esbuild/lib/main.js:1624:15)
at /Users/lgarron/Code/git/github.com/cubing/cubing.js/node_modules/esbuild/lib/main.js:1266:28
at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
errors: [
{
detail: undefined,
id: '',
location: null,
notes: [
{
location: null,
text: 'debug.Stack (runtime/debug/stack.go:24)\n' +
'helpers.PrettyPrintedStack (internal/helpers/stack.go:9)\n' +
'bundler.parseFile.func1 (internal/bundler/bundler.go:186)\n' +
'panic (runtime/panic.go:838)\n' +
'api.importKindToResolveKind (pkg/api/api_impl.go:1595)\n' +
'api.(*pluginImpl).onResolve.func1 (pkg/api/api_impl.go:1637)\n' +
'bundler.RunOnResolvePlugins (internal/bundler/bundler.go:782)\n' +
'bundler.parseFile (internal/bundler/bundler.go:442)\n' +
'bundler.(*scanner).maybeParseFile (internal/bundler/bundler.go:1299)'
}
],
pluginName: '',
text: 'panic: Internal error (while parsing "src/sites/experiments.cubing.net/cubing.js/twisty/simultaneous.ts")'
},
{
detail: undefined,
id: '',
location: null,
notes: [
{
location: null,
text: 'debug.Stack (runtime/debug/stack.go:24)\n' +
'helpers.PrettyPrintedStack (internal/helpers/stack.go:9)\n' +
'bundler.parseFile.func1 (internal/bundler/bundler.go:186)\n' +
'panic (runtime/panic.go:838)\n' +
'api.importKindToResolveKind (pkg/api/api_impl.go:1595)\n' +
'api.(*pluginImpl).onResolve.func1 (pkg/api/api_impl.go:1637)\n' +
'bundler.RunOnResolvePlugins (internal/bundler/bundler.go:782)\n' +
'bundler.parseFile (internal/bundler/bundler.go:442)\n' +
'bundler.(*scanner).maybeParseFile (internal/bundler/bundler.go:1299)'
}
],
pluginName: '',
text: 'panic: Internal error (while parsing "src/sites/experiments.cubing.net/cubing.js/play/index.ts")'
},
{
detail: undefined,
id: '',
location: null,
notes: [
{
location: null,
text: 'debug.Stack (runtime/debug/stack.go:24)\n' +
'helpers.PrettyPrintedStack (internal/helpers/stack.go:9)\n' +
'bundler.parseFile.func1 (internal/bundler/bundler.go:186)\n' +
'panic (runtime/panic.go:838)\n' +
'api.importKindToResolveKind (pkg/api/api_impl.go:1595)\n' +
'api.(*pluginImpl).onResolve.func1 (pkg/api/api_impl.go:1637)\n' +
'bundler.RunOnResolvePlugins (internal/bundler/bundler.go:782)\n' +
'bundler.parseFile (internal/bundler/bundler.go:442)\n' +
'bundler.(*scanner).maybeParseFile (internal/bundler/bundler.go:1299)'
}
],
pluginName: '',
text: 'panic: Internal error (while parsing "src/cubing/search/instantiator.ts")'
}
],
warnings: []
}

This may be because I'm only using the latest package from npm — I haven't figured out how to build a fresher one from the esbuild repo.

Do you have a recommended way one could test out these changes?

@evanw
Copy link
Owner Author

evanw commented Aug 31, 2022

Do you have a recommended way one could test out these changes?

If you want to try out the command line you just need to type make and then ./esbuild <args>. If you want to try out the JS API you can do const esbuild = require('./scripts/esbuild.js').installForTests() to get the current JS API. This is what esbuild's tests do.

@lgarron
Copy link
Contributor

lgarron commented Aug 31, 2022

Do you have a recommended way one could test out these changes?

If you want to try out the command line you just need to type make and then ./esbuild <args>. If you want to try out the JS API you can do const esbuild = require('./scripts/esbuild.js').installForTests() to get the current JS API. This is what esbuild's tests do.

I see. Unfortunately, that doesn't quite work if esbuild is a transitive dependency. Is there a way to replace node_modules/esbuild with something like this, or ideally even an npm link approach that would work? 🤓

@evanw
Copy link
Owner Author

evanw commented Aug 31, 2022

The returned object has a ESBUILD_PACKAGE_PATH property with the package location. You can also just look at the source for installForTests and see what it's doing. It just does npm pack and npm install. You could also try npm link if you'd like.

@arcanis
Copy link

arcanis commented Sep 2, 2022

Will variadic imports be eager or lazy? It'd be quite nice if we could configure that on a case-by-case basis. Webpack does this via comment annotations:

await import(/* webpackMode: eager */ `./icons/${name}.png`)
await import(/* webpackMode: lazy */ `./pages/${page}.tsx`)

(Also very useful, but perhaps out of scope for this iteration, is require.context, which is a Webpack API letting the generated code access the keys that will be accepted by the variadic imports; this way you can easily inject all those modules into a router, webgl textures, other types of caches...)

@evanw
Copy link
Owner Author

evanw commented Sep 3, 2022

Will variadic imports be eager or lazy? It'd be quite nice if we could configure that on a case-by-case basis.

That would be a separate feature, and isn't addressed here. What you are calling "variadic imports" are handled the same way normal dynamic imports are handled. Right now the equivalent behavior for esbuild (in the same chunk vs. a new chunk) is configured with the splitting parameter. In the future I hope to make it possible for plugins to reconfigure which chunk(s) a given module ends up in, but that will require rewriting how esbuild's linker works. I plan to do that in a separate release.

@lgarron
Copy link
Contributor

lgarron commented Sep 4, 2022

The returned object has a ESBUILD_PACKAGE_PATH property with the package location. You can also just look at the source for installForTests and see what it's doing. It just does npm pack and npm install. You could also try npm link if you'd like.

Thanks!

Unfortunately, that fails for me:

Okay, I figured out what broke — it seems to be the use of a plugin:

// build.js (run with: `node ./build.js`)
import { build } from "esbuild";

build({
  format: "esm",
  target: "es2020",
  splitting: true,
  bundle: true,
  outdir: "/tmp/esbuild-test",
  loader: { ".jpg": "copy" },
  entryPoints: ["./entry.ts"],
  plugins: [{
    name: "notify-on-end",
    setup(build) { build.onEnd(() => console.log("onEnd!")) }, // Do almost nothing
  }],
});
// entry.ts
const url = new URL("./cubing.js.jpg", import.meta.url).toString();
console.log(url);
// cubing.js.jpg
(A JPEG file)

It seems that the issue still occurs when the loader config is commented out in the build config, suggesting that the trigger is mainly the use of a plugin + new URL syntax.

> node ./build.js 
✘ [ERROR] panic: Internal error (while parsing "entry.ts")

  debug.Stack (runtime/debug/stack.go:24)
  helpers.PrettyPrintedStack (internal/helpers/stack.go:9)
  bundler.parseFile.func1 (internal/bundler/bundler.go:187)
  panic (runtime/panic.go:838)
  api.importKindToResolveKind (pkg/api/api_impl.go:1595)
  api.(*pluginImpl).onResolve.func1 (pkg/api/api_impl.go:1637)
  bundler.RunOnResolvePlugins (internal/bundler/bundler.go:801)
  bundler.parseFile (internal/bundler/bundler.go:445)
  bundler.(*scanner).maybeParseFile (internal/bundler/bundler.go:1319)

onEnd!
/Users/lgarron/Code/git/github.com/cubing/cubing.js/node_modules/esbuild/lib/main.js:1636
  let error = new Error(`${text}${summary}`);
              ^

Error: Build failed with 1 error:
error: panic: Internal error (while parsing "entry.ts")
    at failureErrorWithLog (/Users/lgarron/Code/git/github.com/cubing/cubing.js/node_modules/esbuild/lib/main.js:1636:15)
    at /Users/lgarron/Code/git/github.com/cubing/cubing.js/node_modules/esbuild/lib/main.js:1278:28
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  errors: [
    {
      detail: undefined,
      id: '',
      location: null,
      notes: [
        {
          location: null,
          text: 'debug.Stack (runtime/debug/stack.go:24)\n' +
            'helpers.PrettyPrintedStack (internal/helpers/stack.go:9)\n' +
            'bundler.parseFile.func1 (internal/bundler/bundler.go:187)\n' +
            'panic (runtime/panic.go:838)\n' +
            'api.importKindToResolveKind (pkg/api/api_impl.go:1595)\n' +
            'api.(*pluginImpl).onResolve.func1 (pkg/api/api_impl.go:1637)\n' +
            'bundler.RunOnResolvePlugins (internal/bundler/bundler.go:801)\n' +
            'bundler.parseFile (internal/bundler/bundler.go:445)\n' +
            'bundler.(*scanner).maybeParseFile (internal/bundler/bundler.go:1319)'
        }
      ],
      pluginName: '',
      text: 'panic: Internal error (while parsing "entry.ts")'
    }
  ],
  warnings: []
}

Node.js v18.8.0

I can also confirm that commenting out our plugin allows esbuild to work in watch mode for our project.

@evanw evanw force-pushed the breaking branch 7 times, most recently from 4089cf8 to 7a44d63 Compare September 4, 2022 13:28
@evanw
Copy link
Owner Author

evanw commented Sep 4, 2022

Unfortunately, that fails for me:

This should be fixed now. Thanks for reporting it.

@lgarron
Copy link
Contributor

lgarron commented Dec 16, 2022

Firefox Nightly is now able to instantiate module workers, which hopefully means that new URL(..., import.meta.url) will soon be valuable for web worker code splitting in all modern browsers! 😄
https://bugzilla.mozilla.org/show_bug.cgi?id=1247687

@lgarron
Copy link
Contributor

lgarron commented Jan 24, 2023

@evanw: If these breaking changes are too much to land in the short term, could I ask/suggest support for import.meta.resolve(...) support as an alternative to new URL(..., import.meta.url)? This is becoming a fairly urgent point of friction with esbuild for any libraries with worker code, because esbuild does not have any out-of-the-box compatibility overlap with other major bundlers/browsers when it comes to such relative path calculations.

Since the time this PR was originally created, import.meta.resolve(...) has now been available in all modern browsers for several months. Unfortunately, code published using import.meta.resolve(...) (e.g. npm packages) will break when used in projects that use esbuild, unless esbuild recognizes the imported path argument as a node in the import graph.

As long as the path argument is handled as part of the import graph, it can be:

  • emitted as import.meta.resolve(...) for esnext targets, or
  • emitted as new URL(..., import.meta.url).href for older ESM targets.
    • (Or into direct dynamic imports, in some cases.)

I'd be happy to send an updated version of this PR if that would help.

@shellscape
Copy link

Chiming in that I'm currently falling over a legacy app using i18n-iso-countries deeply embedded in the dependency chain. @evanw please share some thoughts on status

@quantizor
Copy link

@evanw sorry to pester but would love to get the dynamic import feature in. Is there anything I can do to help?

@guilhermesimoes
Copy link

If I may, I'd like to just comment on the usage of new URL.

We at PeacockTV and SkyShowtime need to support all kinds of TVs and set-top boxes, some of which were launched more than 10 years ago. Some of these are based on older versions of Chromium, and lack URL support.
I know we could add a polyfill but in order to keep our bundle size small we usually try to rewrite our code before adding one. If it would be possible to emit something for older ESM targets that does not rely on URL it would be most appreciated.

@dhdaines
Copy link

dhdaines commented Mar 28, 2023

Hi! I deleted my previous comment because there were ... a lot of different things going on there.

But, in any case, even after working around some problems in Emscripten, this change does not work for loading WebAssembly files that its ES6 module support outputs. You can see a minimal example here: https://github.com/dhdaines/webpack-wasm-test

by running (assuming the breaking branch of esbuild is in your PATH):

npm install
make
perl -i -pe s,wasmtest.wasm,./wasmtest.wasm, wasmtest.mjs
esbuild index.mjs --bundle --outdir=dist-esbuild --format=esm --external:module --loader:.wasm=file --splitting

If you look in dist-esbuild/index.js you will see that it has output this line:

      wasmBinaryFile = new URL("./wasmtest-N3TKMPSX.js", import.meta.url).href;

where the input was:

  wasmBinaryFile = new URL('./wasmtest.wasm', import.meta.url).href;

This is obviously wrong! It will result in the code trying to load the JavaScript wrapper file as WebAssembly.

By contrast, webpack will output the .wasm file with a hash as a name and then load it (depending on your devtool of course) like this:

  wasmBinaryFile = new URL(/* asset import */ __webpack_require__(/*! wasmtest.wasm */ "./wasmtest.wasm"), __webpack_require__.b).href;

There is also an issue with Emscripten's wrapper code doing new URL('./', import.meta.url) which sometimes breaks because both esbuild and webpack try to resolve that as a module rather than just a directory, but that's Emscripten's problem, maybe? Or maybe webpack's? (webpack/webpack#16878)

@dhdaines
Copy link

Basically I think the only reasonable way for a bundler to support new URL(..., import.meta.url) is to try to replicate as closely as possible what the browser would do. It should return a URL that points to the asset that you want to access, not an ES6 module wrapping that URL, unless of course the asset in question happens to be an ES6 module (in which case why would you need to wrap it in a module?)

@dhdaines
Copy link

Should also note that in the case that the target of new URL() is a directory, a bundler should just pass the code through unchanged, since I can't think of any reason why you would want to resolve this to a module import (the browser will definitely not do this, and Node won't either, it's probably a bug that webpack does this).

@AliMD
Copy link

AliMD commented Jun 14, 2023

I love esbuild and I'm waiting for you to officially support ESM capability in worker.
I really don't like having to migrate from ESBuild to another tools like Parcel.
Could you please give a timeline for this feature?
Thank you in advance.

@evanw
Copy link
Owner Author

evanw commented Aug 8, 2023

FYI: The glob pattern part of this PR has been released in version 0.19.0. The documentation for this feature is here: https://esbuild.github.io/api/#glob.

@ggoodman
Copy link

@evanw do you think that the URL syntax for bundling is something that you'll continue exploring at this point or are we waiting for more clear signals from the ecosystem on how to tackle these sorts of non ESM import / export bundle boundaries?

@AliMD
Copy link

AliMD commented Nov 22, 2023

whats up?

@matthieusieben
Copy link

FWIW, I don't think the files referenced using new URL(..., import.meta.url) should be processed in any way. In particular, if the referenced file is a JS, it should not get bundled. The reason for this is that JS files are not the only files that can be referenced this way. Any file can, for any environment. For example, a backend JS code could reference a JS asset for the front-end. These files should not be mixed together (using splitting/bundling).

If, as a user of esbuild, you need to reference another JS bundle from a new URL(..., import.meta.url), then that other file should be bundled on its own, in a separate process.

This would make the use of new URL(..., import.meta.url) more consistent (and simpler to adopt).

@matthieusieben
Copy link

matthieusieben commented Feb 5, 2024

If you do want to support adding referenced JS files to the bundle, then I would suggest to add an argument to the cli:

esbuild app.js --other-entry-point=worker.js

In this scenario, only the new URL('worker.js', import.meta.url) would be bundled (assuming they reference the same file on disk). Any other new URL('my-file.js', import.meta.url) would simply be copied over as a static asset.

ccouzens added a commit to ccouzens/maze-playground that referenced this pull request Apr 14, 2024
This is what I found when looking how to do it:

evanw/esbuild#2508

evanw/esbuild#312
@ArnaudBarre
Copy link

@evanw having this being merge will fix multiple issues in Vite dependency optimizer. We plan to have this working by default on Rolldown, but this is still far from being usable in Vite.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.