[Tracking] Reduce install footprint 🐾

[vanessayuenn]

🧑‍🤝‍🧑 Who: @JReinhold and @ndelangen

This is a tracking issue for the Reduce install footprint 🐾 project. The purpose of this issue is to keep tracking of the overall status of the project and tasks, and plan everything around it.

See also #29072 which is a spike we'll be doing as part of this project to investigate how to move Storybook towards publishing ESM-only packages.

📢 Want to help?

Make sure you read this issue thoroughly to understand what we want to achieve and how.
Do not ask to be assigned to certain tasks, just do them. We won't "reserve" anything to potential contributors, that system rarely work. Help out with what you want, and report findings in this issue, eg. if you've identified potential optimisations in a package. If you open pull requests with your work make sure to reference this issue and tag @JReinhold.

⚠️ Problem

One of the biggest complaints about Storybook is that it's a big and heavy dependency to add to your project. There are muliple ways to interpret the frustration, but one of the most impactful improvements we can make is to reduce the install size of the core Storybook experience.

Throughout the current Storybook 8 releases we've already managed to cut the size significantly, eg. via #27039 and #28519. But there is plenty more work to be done - especially outside the core package - that will enable us to shrink Storybook even further.

🏁 Goals

The high-level goals of this project is to:

1. Reduce the install size of "the core Storybook experience". That is, the packages that are included in a normal init process, such as storybook, the builders (Vite/Webpack), the renderers (React, Svelte, etc.) and the default addons.
2. ... this includes reducing the dependency graph of mentioned packages, to speedup installation.
3. Set long-term baselines and goals that will keep us in check, ensuring we don't regress the install-size later.
4. Build and set up tooling to support these goals. We'll try to find ways that make it easier to maintain lean packages and reduce their footprint. This includes easy-to-access bundle and package analysis tools and reporters.
5. Identify changes that require a major version bump. This project will conclude before the next major version of Storybook, so we can't make those changes now, but we can prepare them so they are ready when we're closer to the next major release.

This means we are not focusing on performance improvements nor reducing the size of a built Storybook. We still love those improvements, but they are not the major focus here.

📚 Resources

• e18e.dev - provides great guides on how to cleanup, speedup and levelup (?😅) packages.
• https://github.com/styfle/packagephobia#how-is-this-different great list of different tools in this space that all achieve different things
• https://github.com/TheDevMinerTV/package-size-calculator
• https://github.com/tinylibs/
• https://github.com/unjs/
• https://github.com/es-tooling/module-replacements/blob/main/docs/modules/README.md

📊 Spreadsheet with package stats

🔬 Methodology

Storybook is a complex mesh of over 30 packages, therefore analysing "Storybook's" install size will be tackled from two angles:

1. E2E tests aka sandboxes: Storybook sandboxes represent real-world, minimal projects with Storybook installed. Storybook support countless project configurations, eg. A NextJS project, or a Vite-based Vue project. By initialising Storybook a top these minimal projects, we get a sense of what impact Storybook has on the overall node_modules size and the amount of dependencies added to the project. The upside of this perspective is that it directly focuses on the experience our users get. The downside of this approach - as with most E2E tests - is that it can be "flaky" because a lot of outside factors can impact the project's install size and dependency count. It's also a coarse metric, detecting significant size increases in sandboxes doesn't really help us identify where the increase is coming from. This brings us to...
2. Unit tests aka packages: Each individual Storybook package will have its own metric on install size and dependency graph. Measuring and detecting size increases at the package-level makes it a lot easier to pinpoint the cause of it. But laser-focusing on packages in isolations can lead to work that makes little difference to the end-user, because they don't use the Storybook packages in isolation. Eg. removing a big dependency from @storybook/react-vite doesn't help if that dependency is still installed by @storybook/core.

Therefore we must keep both metrics in mind, to ensure we improve the experience for the end-user while still getting actionable results on the lower level.

We'll use a breadth-first approach when doing the optimisations across the packages. At the start of the project all, the integration packages have a single task (which also applies to the core): Identify potential optimisations. We'll do that first across all the packages on a high-level, to identify common patterns and find big wins first. This ensures that we don't laser focus on eg. removing 15 KB from @storybook/react when we could instead have removed 12 MB from @storybook/vue3-vite.
With that said, there are already known big wins in the core packages that might get a higher priority than looking into some of the integration packages.

There are many approaches and tools to optimising package's size and dependencies and it's not a one-size-fits-all. https://e18e.dev/guide/cleanup.html is a good primer on how to approach it. At a high-level we'll:

1. Use https://pkg-size.dev to identify and monitor the size of the packages and their dependencies
2. Use https://npmgraph.js.org to identify and monitor the packages' dependency graph
3. Use ESBuild's metafiles that are produced when building all the packages, to identify and monitor the packages' internal structure and bundled dependencies.

We can take @storybook/core as an example:

1. https://pkg-size.dev/@storybook/core@8.3.0-beta.4 shows that there are two major esbuild dependencies significantly contributing to the size. Can we replace it? See Migrate from esbuild to jiti for preset handling #29082
2. https://npmgraph.js.org/?q=@storybook/core@8.3.0-beta.4#select=exact%3Aexpress%404.20.0 shows that express (among others) contribute to a big part of the dependency graph. Can we replace it? See Migrate from express to polka #29083
3. ESBuild's metafiles for @storybook/core (not currently public) shows that Prettier makes up a big part of the bundled output. Can we replace it, or remove it? See Minimize the bundling of prettier in @storybook/core #29084

🚩 Milestones

📈 Baselines and Bechmarks

See spreadsheet detailing all the packages.

Tasks

Beta Give feedback

1. Calculate the packages' own size, dependencies' size and amount of dependencies #29099
   +0
   
2. Calculate the node_modules size diff and lock-file LOC diff after initing Storybook in sandboxes #29100
   +0
   
3. qweqwe

🔧 Optimisations

This section includes all the actual package optimisations that we want to make. The list is highly dynamic and should change a lot during the project.

Architectural changes

Beta Give feedback

1. Minimize the bundling of prettier in @storybook/core #29084
   core dependencies +2
   
2. Use production build of react, react-dom in the manager #29166
   +0
   
3. Investigate removing CJS outputs for client-only entry points #29164
   +0
   
4. Re-establish CJS auto-deduplication with esbuild using cjs-splitting.ts from tsup
5. Scrutinize files included in distributions #29217
   +0

📦 Core Packages

@storybook/core Optimizations

Beta Give feedback

1. Identify potential optimisations
2. Migrate from esbuild to jiti for preset handling #29082
   core dependencies +2
   
3. Migrate from express to polka #29083
   dependencies +1
   
4. CLI: Migrate from chalk to picocolors #28262
   ci:normal maintenance +2
   
5. Core: Migrate from qs to picoquery #28315
   ci:normal core dependencies +3
   
6. [Tracking]: Replace Lodash Usage with Native Functionality or Alternatives #28611
   Tracking help wanted performance issue +3
7. Core: Replace lodash with es-toolkit #28981
   ci:normal maintenance +2
   
8. [e18e]: Replace lodash/merge & lodash/mergeWith with an alternative #28663
   ci:normal dependencies +2
   
9. Migrate from handlebars to simple string concats #29167
   +0
   
10. Upgrade distributed esbuild to match Vite 5's #29252
    +0
    
11. Investigate migrating away from file-system-cache #29168
    +0
    
12. Migrate from fs-extra to node:fs #29104
    +0
13. Core: Replace fs-extra with the native APIs #29126
    ci:normal dependencies +2
    
14. Investigate replacing glob and globby with tinyglobby #29227
    +0
15. Investigate migrating +9 packages to empathic #29103
    +0
16. Investigate migrating from execa to tinyexec #29229
    +0
17. Investigate replacing react-popper-tooltip with @radix/react-tooltip or Floating UI #29161
    +0
18. Investigate migrating away from @yarn/ziplib and friends
19. Investigate replacing or prebundling recast #29143
    +0

create-storybook Optimizations

Beta Give feedback

1. Identify potential optimisations
2. Don't depend on storybook in create-storybook
3. Don't depend on prettier to autoformat the main-config
4. Prebundle dependencies in create-storybook
5. Document npm create storybook instead of npx storybook init in docs

👆 Doing all of the above create-storybook tasks should result in reducing its install size from 73 MB to <1.8 MB and the dependency count from 172 to ~3, greatly reducing the time-to-init.
A counter argument is that a lot of these dependencies comes from storybook which could be globally cached by the package manager. So taking storybook out of create-storybook would just move the download of that package to from pre-init to post-init - but it's just a theory.

@storybook/source-loader Optimizations

Beta Give feedback

1. Identify potential optimisations

🧩 Integration Packages

Builders

@storybook/builder-vite

Beta Give feedback

1. Identify potential optimisations
2. Prebundle dependencies

@storybook/builder-webpack5

Beta Give feedback

1. Identify potential optimisations
2. Investigate getting url out of builder-webpack5 #29181
   +0
3. Prebundle dependencies
4. Discuss making Webpack a peer dependency of builder-webpack5 in 9.0, similar to builder-vite
5. Discuss dropping support for checking types with fork-ts-checker-webpack-plugin
6. Discuss merging all the preset packages into the frameworks (eg. merge @storybook/preset-react-webpack into @storybook/react-webpack5)
7. Discuss merging @storybook/core-webpack into @storybook/builder-webpack5

Renderers

@storybook/react

Beta Give feedback

1. Identify potential optimisations
2. Cleanup dependencies - remove unused and move @types to devDependencies
3. Prebundle dependencies

@storybook/vue3

Beta Give feedback

1. Identify potential optimisations
2. Cleanup dependencies - remove unused and move @types to devDependencies
3. Prebundle dependencies

@storybook/web-components

Beta Give feedback

1. Identify potential optimisations

Frameworks

@storybook/react-vite

Beta Give feedback

1. Identify potential optimisations

@storybook/react-webpack5

Beta Give feedback

1. Identify potential optimisations

@storybook/nextjs

Beta Give feedback

1. Identify potential optimisations
2. Re-use existing babel packages in @storybook/nextjs #29185
   +0
3. Investigate replacing sharp with image-dimensions for detecting the size of AVIF and WebP #29195
   +0
4. Only use tsconfig-paths-webpack-plugin instead of also using tsconfig-paths in @storybook/nextjs #29188
   +0
5. Prebundle possible dependencies in @storybook/nextjs

@storybook/experimental-nextjs-vite

Beta Give feedback

1. Identify potential optimisations

@storybook/angular

Beta Give feedback

1. Identify potential optimisations

@storybook/vue3-vite

Beta Give feedback

1. Identify potential optimisations

@storybook/vue3-webpack5

Beta Give feedback

1. Identify potential optimisations

@storybook/sveltekit

Beta Give feedback

1. Identify potential optimisations

@storybook/svelte-vite

Beta Give feedback

1. Identify potential optimisations

@storybook/web-components-vite

Beta Give feedback

1. Identify potential optimisations

@storybook/web-components-webpack5

Beta Give feedback

1. Identify potential optimisations

Addons

@storybook/addon-docs

Beta Give feedback

1. Identify potential optimisations

@storybook/blocks

Beta Give feedback

1. Identify potential optimisations

@storybook/addon-interactions

Beta Give feedback

1. Identify potential optimisations

@storybook/addon-actions

Beta Give feedback

1. Identify potential optimisations

@storybook/test

Beta Give feedback

1. Identify potential optimisations

@storybook/addon-controls

Beta Give feedback

1. Identify potential optimisations

@storybook/addon-measure

Beta Give feedback

1. Identify potential optimisations

@storybook/addon-outline

Beta Give feedback

1. Identify potential optimisations

@storybook/addon-backgrounds

Beta Give feedback

1. Identify potential optimisations

@storybook/addon-toolbars

Beta Give feedback

1. Identify potential optimisations

@storybook/addon-viewport

Beta Give feedback

1. Identify potential optimisations

@storybook/addon-onboarding

Beta Give feedback

1. Identify potential optimisations

🛠️ Supporting Tooling

Ideally we'd want to set up some tooling that allows to easily inspect package and dependency sizes, to make it easier to optimise these areas. It's still TBD what shape or form these could take.

🌟 Essential tooling

Beta Give feedback

1. Save ESBuild metafiles and make them available from CI #29105
   +0
   
2. Improve readability and usefulness of existing sandbox benchmark reports on PRs
3. Reduce the friction of inspecting packages' build outputs
4. Set up tooling to warn/highlight on regressions in package size and dependency graph #29322
   +0
   

✨ Nice-to-have tooling

Beta Give feedback

1. Add eslint-plugin-depend to check all packages' dependencies #29251
   +0
   
2. Reduce the friction of inspecting packages' dependency graph
3. Add knip to identify unused files/exports #29106
   +0
   

Related: evanw/esbuild#3909

🤷 Misc

🎁 Wrap up

Beta Give feedback

1. Create a new tracking issue containing the known optimisations we didn't had time to do in this project
2. Compile a list of optimisations we want to do for 9.0 👇
3. Publish a blog post about the reduced footprint

💥 Breaking optimisations

Beta Give feedback

1. Stop importing all of @storybook/icons in @storybook/core #29159
   BREAKING CHANGE components core +3
2. Investigate replacing react-syntax-highlighter with shiki #29160
   +0

💖 Nice-to-haves

Beta Give feedback

1. Investigate the impact on running storybook dev at the end of storybook init
2. Remove @storybook/addon-links from the default addons that are part of init #29073
   +0

[JReinhold]

Based on the baseline calculations, I think it's fair to skip the following packages along with @storybook/addon-essentials for now, because they are "tiny" compared to the rest of the packages here:

[TheThing]

Just a small heads up, I think a rogue file might have snuck its way into the release tarballs ;)
I downloaded the latest tarball release (8.3.2) straight from npm and there seems to be a rogue __mocks__ that snuck through. Not a big deal but I figured I should report it :)

[pumano]

Maybe https://github.com/ai/size-limit helps with process to automate code + deps size on pull requests

[ravicious]

If I can chime in, #25387 already moved @types/node from a dep to a dev dep in core-common, but I see that @storybook/react still includes @types/node as a dep. #24873

See also SBoudrias/Inquirer.js#1569. This was the dep that caused msw to include @types/node as a dep. They've recently fixed that issue and msw was updated not to do that. When upgrading msw, I realized that @storybook/react is another package in our package.json that does this.

[JReinhold]

@ravicious I believe that has already been fixed?

https://github.com/storybookjs/storybook/blob/next/code%2Frenderers%2Freact%2Fpackage.json

[ravicious]

Oh, my bad. I didn't notice it because I've been looking at 8.3.6 on npmjs.com. I haven't looked at the current version in the repo.