support for top level await

[buttercubz]

I am currently working with deno which has top level support, is there any way to use esbuild with top level await?

[evanw]

Sorry, top-level await is not supported. It messes with a lot of things and adding support for it is quite complicated. It likely won't be supported for a long time.

[guybedford]

Out of interest / for the issue history, is it possible to give some idea of the sort of issues that arise here?

…

On Sun, Jul 12, 2020 at 14:36 Erick Sosa Garcia ***@***.***> wrote: Closed #253 <#253>. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#253 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAESFSS2NEUIRROFVJVY7GLR3IUGHANCNFSM4OX6XTIQ> .

[evanw]

Right now there is an equivalence in esbuild between CommonJS modules and ES6 modules. A module can be written in either syntax and can even mix both syntaxes. That no longer works with CommonJS because the implementation of require() can't suspend execution in JavaScript. So I'll have to figure out something for that. It's probably some form of conditionally disabling features along the module dependency graph when a module contains a top-level await.

I'd like to punt this feature until later because there are already a lot of edge cases to work through with all of the features added so far (tree shaking, scope hoisting, code splitting, star re-exports, etc.) and this feature adds another dimension to the test case matrix. This feature is also lower priority for me than other features such as watch mode, plugins, finishing code splitting, and adding additional CSS and HTML file types, especially given that top-level await is a new feature and not yet widely used.

Let's keep this issue open because I'm sure it'll come up again. At the very least esbuild should be able to parse top-level await and pass it through in non-bundling mode, which will be useful for people using esbuild as a transformer and not a bundler.

I haven't added parsing support for top-level await yet because it's not clear exactly how to parse it. The way it works the JavaScript spec is that you're supposed to know beforehand whether the parsing goal is "script" or "module" and then only parse the await operator with the "module" goal. However, esbuild doesn't work that way because it makes no distinction between the two goals.

TypeScript is another tool that does this so I've been waiting to see how they handle things. When I looked into this last they had a pretty severe bug so I'm going to wait a while longer for their implementation to settle before I consider how to approach this in esbuild. It looks like they still have open bugs on this such as this one, which interestingly appears to require a symbol table to parse correctly kind of like C. Not sure how I feel about that approach.

Also, while I'm on the topic, apparently this feature also requires supporting top-level for await loops. Just adding that as a note to myself for when I get to implementing this.

[guybedford]

That no longer works with CommonJS because the implementation of require() can't suspend execution in JavaScript

This was one of the major issues that the Node.js integration of ES modules had to deal with. The result was we did not permit CJS modules to load ES modules although bundlers like Webpack do. I believe previous dicussions here for eg Webpack have been around allowing require of ES modules so long as they do not use top-level await. Because of the timeline for adoption of top-level await, we may actually be ok here in terms of real world usage.

I haven't added parsing support for top-level await yet because it's not clear exactly how to parse it. The way it works the JavaScript spec is that you're supposed to know beforehand whether the parsing goal is "script" or "module" and then only parse the await operator with the "module" goal. However, esbuild doesn't work that way because it makes no distinction between the two goals.

This was something else the Node.js modules integration had to carefully define with "type": "module" package.json boundaries. Mainly for the strict-mode-by-default handling for modules without module syntax, but also for other reasons like this too.

I would avoid trying to follow anything TypeScript does personally as they have made it clear they will not lead the ecosystem but instead follow the winning conventions when they emerge. As such TypeScript is a typical friction point in modules workflows these days.

One of the other implementation difficulties is that when bundling parallel dependencies, they are supposed to behave like a Promise.all, but in theory that is just a matter of wrapping.

Definitely agreed this is a low priority, but thanks for the detailed explanation and for keeping this open for future reference, however long that might be.

[evanw]

I'm planning to add pass-through support for this now that node is enabling it by default. It won't work while bundling but it will still be useful for certain use cases such as using esbuild as a library for converting TypeScript to JavaScript.

TypeScript is another tool that does this so I've been waiting to see how they handle things.

It looks like TypeScript's implementation ended up being to do an initial parse where top-level await is an identifier, then to reparse all expressions containing those identifiers as await expressions instead if the AST has an import or export statement. This approach isn't a good fit for esbuild's parser for many reasons including lack of a persistent token stream and the fact that parse errors are streamed in real-time and cannot be un-emitted. It's also a very complex approach because it may involve rewriting subsequent expressions (e.g. await \n foo; becomes await foo; instead of await; foo;). And it seems too weird to me to not be able to use await import() in a file without having to add export {} like you have to in TypeScript.

I've decided to just try unconditionally parsing the await operator at the top-level. I'm guessing that the number of modules in the wild that actually expect await to be an available top-level identifier is very small and the benefit of supporting top-level await syntax far outweighs making those modules work, especially for a modern tool like esbuild.

[MylesBorins]

@evanw an added benefit is that TLA is only supported in the module goal and await is a reserved keyword in the module goal 🎉

[Soremwar]

Note that while transforming code containing top-level await is supported, bundling code containing top-level await is not yet supported.

:(

Is the situation for this still the same?

[evanw]

Is the situation for this still the same?

Yes, top-level await still doesn't work with bundling.

[evanw]

TLA is so complicated. I'm still trying to figure out how to support it but it has been really challenging. I'm trying to match all of the real evaluation order rules including parallel evaluation and microtask ordering while still doing scope hoisting and my attempts keep failing. I might need to undo some of esbuild's existing optimizations to get this in.

One of the other implementation difficulties is that when bundling parallel dependencies, they are supposed to behave like a Promise.all, but in theory that is just a matter of wrapping.

It would be amazing if that actually worked but joining together modules using promises causes things to evaluate in an incorrect order. I have written a TLA correctness fuzzer to verify that a TLA bundling strategy is correct by comparing it against V8, which I assume is correct. The only strategy that I've gotten to work is manually creating a module registry and using it to track which dependencies are remaining for each module. Presumably this is how the runtime works as well. But that's too much generated code size and run-time performance overhead for what I'm going for.

I haven't quite gotten to the bottom of exactly what's so hard about this, but I have a hunch that the problem might be a fundamental difference between how the spec works and how promises work. I think it might be something like "the spec adds dependencies as modules are evaluated but the Promise API adds dependencies at the time .then() is called." I'm still investigating this.

FWIW I couldn't get Webpack or SystemJS to match V8's behavior either. Does anyone know if there is a bundler-like implementation of TLA that is able to match V8's behavior? Also before I get even deeper into this, does anyone know if V8's behavior is incorrect somehow?

[evanw]

Small update: It turns out that the top-level await specification and/or V8 have some subtle bugs that cause undesirable behavior. More details are here: evanw/tla-fuzzer#1. Many thanks to @sokra on the Webpack team for getting to the bottom of these issues.

[evanw]

The newly-released version 0.10.0 has preliminary support for bundling with top-level await (previously top-level await was not allowed at all when bundling). From the release notes:

• Initial support for bundling with top-level await (support for top level await #253)

  Top-level await is a feature that lets you use an await expression at the top level (outside of an async function). Here is an example:

  let promise = fetch('https://www.example.com/data')
  export let data = await promise.then(x => x.json())

  Top-level await only works in ECMAScript modules, and does not work in CommonJS modules. This means that you must use an import statement or an import() expression to import a module containing top-level await. You cannot use require() because it's synchronous while top-level await is asynchronous. There should be a descriptive error message when you try to do this.

  This initial release only has limited support for top-level await. It is only supported with the esm output format, but not with the iife or cjs output formats. In addition, the compilation is not correct in that two modules that both contain top-level await and that are siblings in the import graph will be evaluated in serial instead of in parallel. Full support for top-level await will come in a future release.

[dtruffaut]

It works and this is awesome !
Now I can use const { a, b } = await import('c'); on top of all my modules !
It saves lots of duplicates and increases readability :)

[Lcfvs]

Hi,

Into my project, I experimented an issue with a top-level await, on the 0.11.13 version.

Actually, esbuild still let some await on the top-level of the bundle (--format=esm).

I solved it by adding a wrapper like --banner:js='(async () => {' --footer:js='})()'.

Is there a reason to don't wrap the entire bundle, by default (or just if it have some top-level await)?

If the project sources can help: https://glitch.com/edit/#!/immutable-isomorphic-demo?path=package.json%3A7%3A53

(The problematic await is located into the dist/worker.js bundle and the expression to find await dependency('renderer'), originally into the templates)

[evanw]

Is there a reason to don't wrap the entire bundle, by default (or just if it have some top-level await)?

Yes. Doing that would be incorrect. Top-level await is supposed to delay the evaluation of the importing module until the promise has been resolved, and the only way of doing that is to use a top-level await. In other words, top-level await cannot be polyfilled for environments that don't have it, at least as long as you support the possibility that the output file may be imported by something else.

[Lcfvs]

Yes. Doing that would be incorrect. Top-level await is supposed to delay the evaluation of the importing module until the promise has been resolved, and the only way of doing that is to use a top-level await. In other words, top-level await cannot be polyfilled for environments that don't have it, at least as long as you support the possibility that the output file may be imported by something else.

Ok for that, but since the ES modules aren't supported into a worker, the top-level await can't be supported.

Maybe need a --platform=worker to handle a specific behavior and considerations? :) (or another flag to enforce that wrapping, whatever the context)

[evanw]

Ultimately top-level await will be supported with the iife format, which is really what you're trying to do. It will work similarly to --banner:js='(async () => {' --footer:js='})()' but that should be a suitable substitute for now.

[dtruffaut]

ES modules aren't supported into a worker

Yes they are, in both worker and service worker.
Use the {type: module} option when instanciating a worker or a service worker.

https://blog.chromium.org/2021/04/chrome-91-handwriting-recognition-webxr.html (look at bottom of the page)

ES Modules for service workers ('module' type option)
JavaScript now supports modules in service workers. Setting 'module' type by the constructor's type attribute, worker scripts are loaded as ES modules and the import statement is available on worker contexts. With this feature, web developers can more easily write programs in a composable way and share them among a page and workers

[Lcfvs]

Yes they are, in both worker and service worker.
Use the {type: module} option when instanciating a worker or a service worker.

https://blog.chromium.org/2021/04/chrome-91-handwriting-recognition-webxr.html (look at bottom of the page)

ES Modules for service workers ('module' type option)

Unsupported, actually, on Firefox ;)

[Nantris]

@ctjlewis when trying your suggestion:

Invalid build flag: "--module=esm"

I found --format=esm maybe works.

[ctjlewis]

Yeah, that flag is now called —format. I’ll update the answer shortly.

ESM target appropriately includes top-level await because ES Modules allow top-level await. Just saw your edit.

I’m going to be running ESM through pkg soon and I’ll add notes here.

[RangerMauve]

@ctjlewis Did you manage to get ESM working with pkg? I've been trying to get it to at least run via dynamic import() calls, but then the source files don't seem to be getting loaded for it.