[Tracking] ESM-only spike 📦

[JReinhold]

🧑‍🤝‍🧑 Who: @ndelangen and @JReinhold

This is a tracking issue for the ESM-only spike 📦. The purpose of this issue is to keep tracking of the overall status of the project and tasks, and plan everything around it. It is a sub-project under #29038

The main purpose of the spike is to answer questions about what happens when we convert to ESM-only, and how that can happen. Answering some of these questions will require explorations and writing code. It is not the goal of this spike to merge the code into next, even if it backwards compatible. However it could potentially be part of v8.5. The reason for this is that we always underestimate the effort it requires to turn an exploratory spike-PR into a polished PR ready for merging.

As part of this we need to ensure that we're testing in a proper setting. Sometimes testing these far-reaching changes isn't enough in the monorepo, because the monorepo can have linking setup that makes the changes work, even though they wouldn't in a real setting. We must be mindful about when things should be tested in separate projects using Verdaccio or similar.

Tasks

Beta Give feedback

1. Write down how to figure out the answers to the questions @ndelangen

❓ Questions

What are the steps towards ESM-only, and in what order should they be done?

How can the migration be split into multiple milestones to avoid massive, unreviewable PRs?

Investigate if we can gradually turn each entry point in @storybook/core to ESM-only, without doing it for the whole package at once. What happens when we do this? We've already started looking into this in #28797

Here's another spike proving it's feasibility:
https://github.com/storybookjs/storybook/pull/29320/files

Which of the steps can be carried out in a backwards-compatible way, potentially pre-9.0?

The CLI package is already type=module. However the bin file is explicitly a .CJS-file.
It's relatively simple to change this to either be ESM directly, or by writing an import() there, and load the dist code.

A spike can be found doing just that here:
#29317

What is the right order of package migrations that will minimize failures?

How can we load presets?

Today we’re using esbuild-register, but that’s a hurdle because it assumes CJS-first - can we replace that with something else, like esbuild natively? How do we keep supporting TS, CJS and ESM presets?

In #28796 we investigated replacing esbuild with jiti, and found that it was more strict in which CJS/ESM files it could parse. We should investigate jiti more closely and "stress-test" it with different preseet bundles and main.js/cjs/mjs to see how it will break existing use-cases.

In #28802 we investigated using esbuild directly instead of esbuild-register. We'll then bundle and write all presets to a temp dir with the .mjs extension to be imported. We'll continue down that path here. We should also investigate how Vite loads its config files, because they also use esbuild for this, so we can get inspiration from their logic.

Is bottom-first or top-first the best approach for migration?

1. bottom-first would be migrating the Storybook core to ESM, and make all other dependents import it via dynamic imports to force ESM usage. Then slowly move up the dependency graph, migrating one package at a time.
2. top-first would be migrating the bins to ESM, making Node.js go into "ESM mode", and then see what happens.

To be clear, in this paradigm, the logic would be like this, top-to-bottom:

1. Storybook CLI / bin
2. Builders
3. Renderers / frameworks
4. Preset loading
5. Storybook Core

Which use-cases, environments are we dropping support for by going ESM-only. Specifically, what about Jest-based Portable Stories users? Or anything else?

We know that Jest in general is very heavy on CJS. Does going ESM-only break Jest-usage of Portable Stories, or is it still compatible? The upcoming release of Jest 30 might improve the situation. We might reach a conclusion that involves a recipe for how to make ESM-only Storybook work in a Jest environment.

How will this affect React Native?

Step 0 here would be to have a set up that allows us to regression test a simple React Native project against the changes we're making in the spike

• We're also seeing requires of @storybook/core which would break in an ESM-only world: https://github.com/storybookjs/react-native/blob/next/packages/react-native/scripts/loader.js#L5
• Maybe we need a React Native specific entry point in core that has all the utilities that package needs, in CJS?

[JReinhold]

Here is where Vite bundles the config file:
https://github.com/vitejs/vite/blob/5e56614ccef9b5a585efef27a3dca9e05b87615e/packages/vite/src/node/config.ts#L1545-L1694

and then that result is passed to https://github.com/vitejs/vite/blob/5e56614ccef9b5a585efef27a3dca9e05b87615e/packages/vite/src/node/config.ts#L1701-L1745

It's all done in
https://github.com/vitejs/vite/blob/5e56614ccef9b5a585efef27a3dca9e05b87615e/packages/vite/src/node/config.ts#L1475-L1543

[ndelangen]

• the cli package is already type=module
• the core package is already type=module
• making the CLI package actually use it's own ESM output, is relatively simple:
  ESM: CLI ESM-mode runtime #29317
• The core package already has a hack to ensure the ESM bundle will contain references to require, __dirname and __filename, to support faux-ESM source files.

[valentinpalkovic]

How can existing Storybook addons, which still produce both CommonJS (CJS) and ESModule (ESM) build assets (or even only CJS), be configured to properly consume a pure ESM-only @storybook/core package? What strategies or best practices can be applied to ensure compatibility, considering that these addons currently support both module formats? Specifically, how can we handle the transition for addons still relying on CJS, while moving the core package to ESM-only?

[JReinhold]

How can existing Storybook addons, which still produce both CommonJS (CJS) and ESModule (ESM) build assets (or even only CJS), be configured to properly consume a pure ESM-only @storybook/core package? What strategies or best practices can be applied to ensure compatibility, considering that these addons currently support both module formats? Specifically, how can we handle the transition for addons still relying on CJS, while moving the core package to ESM-only?

@valentinpalkovic that's something we probably want to investigate as part of investigating preset loading.

Given Storybook loads the addons (and not the other way around), I'd expect that only the ESM entrypoints would be used, and CJS being ignored. However today some addon entry points are only built in CJS, like preset entry points, and that might be a challenge to support. In those cases I'd expect addon authors would need to release new ESM-only versions of their addons (with our guidance). But we'll see.