Publish Blockly as ES modules

[cpcallen]

Check for duplicates

• I have searched for similar issues before opening a new one.

Problem

At the moment much of our documentation and examples (including most of the code in the blockly-samples repository) load Blockly by doing

import * as Blockly from 'blockly'

as if Blockly were published as ES modules—but in fact the blockly NPM package contains only UMD modules which are (in practice) either require()ed as CJS modules or simply loaded as <script>s.

Certain JavaScript runtimes—notably node.js—can deal with this, allowing developers to load UMD modules (as CJS modules) using the import keyword, but no browser supports this. This means that none of the examples we publish can be run without some kind of bundler, such as webpack or the Closure Compiler, or other build step, to combine and/or convert the developer code and Blockly modules into a compatible form.

Request

1. Make it possible to import Blockly in a browser without needing a build step, by publishing Blockly as an ES module (or, more accurately, as one ES module for each chunk: blockly, blocks, javascript etc.) Although import … from 'blockly' cannot work in a browser without an import map, it should be possible to import * as Blockly from './path/to/blockly.js';` (or '…/blockly.mjs', depending on choices discussed below).
2. Continue, for the time being at least, to support loading Blockly as a <script> and/or as a CJS module via require().

Alternatives considered

The NPM format allows publication of hybrid ESM+CJS modules. This is done by having separate "main": and "module": entries in the package's package.json file, pointing at the CJS and MJS entry points for the package respectively. For subpackages such as blockly/blocks there is an "exports": directive supported by node.js and by webpack, but this is not part of the official NPM package.json format.

There are at least two possible approaches to how such hybrid packages work, however:

1. The package can contain two completely separate copies of each entry point module: one built as an ES module and the other as a CJS module. Here is a description of how such a dual-binary package might be created.
2. The package can each entry point module built as a CJS module, and an thin ESM wrapper that loads the CJS module and re-exports its exports. This is roughly akin to what the loading shims introduced in PR refactor(tests): Introduce loading shims, use in playgrounds #7380 do when loading Blockly in compressed mode. This article explains why this approach might be preferred and details about how to implement it (along with much additional background information about the ESM vs. CJS situation).

Additional context

• Although the UMD format allows one to publish a single file that can be treated as a plain script, as an AMD module or as a CJS module, there is no way for a file to be simultaneously both a UMD module and also an ES module—at least, not if it has any exports, because the export keyword is only legal in an ES module, and it is not possible to export anything from an ESM without using the export keyword. This means that there must be at least two entry point files present (one for ESM and one for UMD) in any package if it is to support being loaded both with import and require().

• This 2ality article provides some additional (though partially outdated) information about different ways to structure a hybrid ESM+CJS NPM package.

• Although we should continue to support loading Blockly as a CJS module for the time being—at minimum to provide a substantial transition period in which developers can move to using the ESM version—we may well wish to ultimately ship the blockly NPM as ESM-only. There may be some value in continuing to support CJS in the long term, but UMD's abilito to allow one to load Blockly via <script> is largely unnecessary when all current browsers now support <script type="module"> import * as Blockly from './path/to/blockly.js';.

[jogibear9988]

Are there plans for this to be solved in the next months?
I'm currently workin on a UI Designer (https://node-projects.github.io/web-component-designer-demo/index.html) wich uses no bundling at all. This UI Designer is used as a UI Designer for a homeautomation system (iobroker), in wich I'd like to include blockly as a scripting language (in the frontend, backend already has blockly). But as my designer uses no build step at all, ESM Modules would be much easier to include.

[cpcallen]

We anticipate publishing a public beta of Blocky v11 that will include ESM entry points early in 2024 Q1, probably in January.

Caveats: not a promise, not necessarily in the first v11 beta, and no guarantees about when v11 (non-beta) will be published (though probably before the end of Q1, possibly as early as mid February.)

[BeksOmega]

Realized on Friday that this might be blocked by some monkey patches we did to get around circular dependencies. See original context doc [edit: this is internal and cannot be shared to non-googlers :/ ] as well.

@cpcallen could you take a look and see whether this is a blocker or not?

[cpcallen]

Realized on Friday that this might be blocked by some monkey patches we did to get around circular dependencies.

No, that's fine. Monkey patching like this is kind of gross, but .prototype objects—even ones created via class rather than function—are mutable so this works perfectly well in a 100% ESM (or TSM) world.

[jogibear9988]

I've created a new issue: #8170 cause this ESM modules are not usable in the browser

[cpcallen]

This issue was closed prematurely. PR #8091 introduced ESM entrypoints, but we do not yet publish Blockly as (browser-ingestible) ES modules.

State of Blockly packaging as of v11.0.0:

• In 2022 most of core/ was migrated from JavaScript (with goog.modules) to TypeScript (with TS modules, which are equivalent to ES modules), with blocks/ and generators/ being completed in 2023.
• The main build pipeline remains unchanged:
  • The .ts sources are compiled by tsc into corresponding .js files (which are ESM, despite our top-level package.json not having "type": "module"), which are then chunked, bundled, and minified by Closure Compiler, into blockly_compressed.js, blocks_compressed.js, javascript_compressed.js etc.
  • These chunks use a UMD wrapper to allow them to be consumed as CJS (using require()), AMD (does anyone do this anymore?) or loaded via a <script> tag (in which case they create global variables in order to supply their exports to other code).
• In PR feat(build)!: Introduce exports section in package.json #7822 we added an exports section to package.json that points the blockly/core, blockly/blocks, blockly/javascript (etc.) entrypoints directly to these bundles, as the "default" conditional export.
  • This replaces the wappers (core.js, blocks.js, javascript.js) that previously appeared at the top level of the Blockly package.
  • We initially deleted these wrappers, but feedback from beta users alerted us to the fact that Browserfify does not support the exports directive package.json, so the wrappers have been retained for the time being. (I note that it seems Browserify may not support ESM at all, so developers using it will probably need to find an alternative in any case.)
• In PR feat(build)!: Introduce ESM entrypoints #8091 we added "import" conditional imports pointing at new ESM wrappers (blockly.mjs, blocks.mjs, javascript.mjs etc.) that import the corresponding *_compressed.js bundle and reexport its named exports.
  • These wrappers depend on the consumer (JS runtime or build tool) being able to import CJS modules. This is true for Node.js and webpack at least.
  • In a sense this is not a meaningful change from the previous state of affairs, where developers needed to use such a tool in order to import Blockly into their app. However, the presence of "import" conditional exports in package.json is an important intermediate step towards the end goal of publishing Blockly as ES modules and a prerequisite to pursuing any of the options discussed below.

Options for further modernisation of the Blockly package:

Status quo: UMD with node.js- and bundler-compatible ESM wrappers

This is the do-nothing option, sacrificing the possibility of importing Blockly in a browser (absent bundler or other build steps).

• It is already possible to load blockly_compressed.js et al. as <script>s, so perhaps being able to import it is not so useful.
• On the other hand, almost of our documentation and examples use import, so this makes it more challenging to get started using Blockly without a build process—and it should not be necessary to have a build process set up to get started, even if our create-package script makes that set-up relatively easy.

Separate CJS and ESM builds

We could provide separate CJS and ESM builds of each of the entrypoints. The simplest approach would be to modify our build pipeline to (e.g.) create both blockly_compressed.js and blockly_compressed.mjs, and update package.json have the "import" conditional export for blockly/core point to the latter.

This has the advantage that the .mjs modules would be "proper" ES modules and therefore be directly importable by browsers, either via a full relative path or via an import map (as in this example that @jogibear9988 provided for #8170).

The significant drawbacks of this approach are:

1. The dual-package hazard, where the CJS and ESM builds do not share state. This would manifest itself most immediately in that Blockly plugins, which are currently compiled to UMD (i.e., CJS) modules using webpack, would mostly be broken for developers who use import, since (e.g.) the field plugins would register their new field types with the CJS copy of Blockly while the application would be trying (unsuccessfully) use them in the ESM copy of Blockly. We could update our plugins similarly but this could still be an issue for third-party plugins.

   There are some potential workarounds to this, mainly by making sure that all of Blockly's (extensive) global state is stored in some place that it can be accessed from both the CJS and ESM copies of Blockly. This could be done using global variables, or (in principle) by keeping it all in a shared module that could somehow be loaded by both the ESM and CJS copies of Blockly—but that is just begging the question.

2. That projects that use both import and require to load Blockly (including inadvertently, e.g. via plugins) would end up containing two full copies of Blockly, causing undesirable page bloat.

Make the ESM wrappers browser-compatible

…via require polyfill

It is in principle possible to provide a require polyfill that can run in a browser—for example, the require-browser package provides such, albeit in an explicitly not-suitable-for-production form. The ESM wrappers could then be changed from:

import Blockly from './blockly_compressed.js';
export const {ASTNode, BasicCursor, /* ... */} = Blockly;

to

if (typeof require !== 'function') {
  globalThis.require = function(path) { /* Browser-compatible require polyfill. */ };
}
export const {ASTNode, BasicCursor, /* ... */} = require('./blockly_compressed.js');

(The require implementation doesn't even need to be fully general, just sufficient for this particular application.)

This approach risks replacing the dual-package hazard with an equivalent multiple-require hazard:

• One of the main features of require is that it loads each module at most once, so calling require('./path/to/file.js') to the same file (even if referred to via a different path e.g. due to different base directory) always returns the same (===) JS object.
• This necessitates some kind of module cache, that must be shared between all invocations of require().
• That would be easy enough to do for the inline require implementation above (probably via global variable), and we could provide compatible wrappers for the plugins in blockly-samples that also (in their compiled form) require Blockly, but there's no general approach that will work for third-party Blockly plugins.
• The require polyfill would have to be very carefully tested with many different build tools, so that developers who were using a bundler would continue to have their builds work in the way they have until now. It would be bad if the bundle ended up using both the bundler's own version of require (however implemented) as well as our require polyfill, as they would not share a common module cache and so blockly_compressed.js would be likely to be loaded multiple times even if only one copy were included in the bundle.

…via loading *_compressed.js as scripts

An alternative to providing a require polyfill would be to have the ESM wrappers load the corresponding *_compressed.js as a script. This is the approach taken by the playground loading shims, which (when loading in compressed mode) use a helper function to load the compressed chunk as as a <script>.

• This would be slightly simpler than providing a full require implementation.
• It does result in setting global variables Blockly, libraryBlocks, javascript etc.
• To avoid the dual-package hazard we'd still need to avoid duplicating Blockly's shared state. This would probably most easily done by having the UMD wrappers set those same global variables even when the chunk was loaded as a CJS (or AMD) module—but first checking to see if the global was already set and returning existing value if so. This would make Blockly behave as if it were an old-fashioned goog.provide / TS namespace script, albeit one dressed in fancy clothes.

Provide only ESM entrypoints

We could publish Blockly as ES modules, and drop the CJS build and entrypoints completely. This would be a breaking change.

Creating a CJS wrapper for an ES module isn't really possible due to ES modules being loaded asynchronously; at best the CJS wrapper could return a Promise that returns the Blockly module (exports) object, and this large difference in behaviour would be very confusing. It's probably better that pure-CJS client code use the dynamic import() function directly instead.

There are a few questions to decide:

1. Do we publish it in the from of individual modules (i.e. the separate .js files currently emitted by tsc), or as a single rolled-up ESM per chunk? In favour of publishing it as individual modules:

   • This could allow us to decommission most of our build tooling and (with some minor organisation of the repository) also most of our packaging scripts.
   • It would make debugging easy (even absent sourcemaps).
   • It offers (potentially) the smallest final bundle sizes for developers who have their own build tooling, since it makes tree-shaking as easy as possible.

   In favour of rolled-up chunks:

   • For developers without a build step, this will load much faster. (We have noted that loading the playground uncompressed locally is fine, but loading it uncompressed over the network—e.g. when testing against github pages / app engine—can be painfully slow due to the large number of files.)
   • This would make it possible to at least attempt to enforce @internal restrictions on non-top-level exports.

2. Do we want to continue to supply blockly_compressed.js et al. for use when loading Blockly via a <script> tag (either locally or, as a few of our documentation examples do, from unpkg.com)?

   • Since most browsers now support <script type="module"> this is much less necessary than it used to be.
   • It may be that removing it will break some websites that have been using unpkg (though they could update to use import or pin to an earlier version).
   • We could optionally remove the UMD wrapper and make these bundles be plain scripts (as they were a long time ago), to make their intended use very clear.

Recommendations

My view is that (as of 2024) the world is pretty clearly moving in the direction of ES modules, and the need to support CJS is quickly diminishing. We should definitely solicit feedback from our external developers (particularly major partners); to do so I propose that (at a suitable juncture, possibly after we have migrated Blocly to a monorepo) we publish one or more experimental (alpha?) versions of Blockly packaged as pure-ESM. This would give us a chance to see what the changes to our build pipeline actually look like, and allow us to get feedback from external devs about metrics they might care about (download size, load time, ease of upgrade, etc.)

If developer feedback is positive then at the (then) next major version we should drop CJS support.

[cpcallen]

Marked triage so remaining work can be prioritised appropriately.

[jogibear9988]

I would vote for the complete ESM switch. If someone needs CJS or old Code, it's part of the application to transpile, not of the library.
I work on a designer (https://node-projects.github.io/web-component-designer-demo/index.html) wich works completely without any build (if you don't count typescript, but for this you could switch to .js with jsdoc), but I've problems with blockly cause it is not ESM. They main problem here is Monaco Editor is also not ESM, wich then adds it's own require, and if you then import blockly after monaco, it's detection if it should load global fails.
So switch to ESM would be best solution (cause I think monaco will need much more time)

[cpcallen]

They main problem here is Monaco Editor is also not ESM, which then adds it's own require, and if you then import blockly after monaco, it's detection if it should load global fails.

Somewhat tangential to this bug, but as I'm not familiar with Monaco: can you explain what it is doing ("detection if it should load global fails") and how Blockly being published as ESM would fix that?

[jogibear9988]

They main problem here is Monaco Editor is also not ESM, which then adds it's own require, and if you then import blockly after monaco, it's detection if it should load global fails.

Somewhat tangential to this bug, but as I'm not familiar with Monaco: can you explain what it is doing ("detection if it should load global fails") and how Blockly being published as ESM would fix that?

Sure: here is a sample wich loads monaco and blockly. https://github.com/jogibear9988/blocklyEsmTest/blob/main/monaco.html
It loads monaco as stated in their tutorial.

After that blockly is not able to load:

[jogibear9988]

I could fix this, if I wait until monaco is loaded, then delete the "require" and than start to load blockly....

But it's bad, sure also monaco should provide ESM modules. But it is bad that the modules do different thinks, depending on wich globals variables are set

It this part wich breaks:

  if (typeof define === 'function' && define.amd) {
      // AMD
      define([], factory);
  } else if (typeof exports === 'object') {
      // Node.js
      module.exports = factory();
  } else {
      // Script
      root.Blockly = factory();
  }

I prefer real ESM. There I know always what it does. It does not pollute any global namespace.

[cpcallen]

I could fix this, if I wait until monaco is loaded, then delete the "require" and than start to load blockly....

Oh yes, I see what's going on. Deleting require would work; alternatively, since our UMD wrappers treat the existence of define as an indication that they're being loaded as AMD modules, you could actually load them that way (i.e., using require.js's require, as you do for the editor), instead of loading them using <script> tags.

I'm going to describe this behaviour as "working as intended", though certainly Blockly being ESM-only would also fix your issue. (We've also discussed removing support for AMD in particular, even if we kept script + CJS support, since we have the impression no one uses it any more; it seems however that your example provides at least a partial counterexample.)