Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build Tooling: Explore faster builds split into individual packages #37318

Closed
wants to merge 15 commits into from

Conversation

youknowriad
Copy link
Contributor

@youknowriad youknowriad commented Dec 13, 2021

Related to #36142.

This is not necessarily meant to land as is but more an exploration to see what's possible in the future.

My main motivation here was that the JS community has been evolving in the last couple years, there are new approaches, new toolings... and I wanted to see how other folks approach monorepos what makes sense for us to adopt. Some of the tools I've looked at:

Some of my own conclusions/notes:

  • Basically everyone is moving towards faster Rust/go based compilers and tools.
  • Babel is on the way out
  • Our own way to build packages and do monorepo handling is very adhoc and a bit different from the industry standards of the moment.
  • It seems pnpm is slowly becoming the goto package manager for mono-repos because of its smart and efficient node_modules folder management.
  • Changelog handling in changesets is very cool, in additions to handling the npm publishing and changelogs, we could take inspiration from it to handle our own plugin release changelogs.

Initial exploration in this PR:

In this particular PR, I trying seeing whether we can use one of the two most popular Rust/go based JS compilers instead of babel for our packages (swc or esbuild, I've tried both and used swc for now).

The first take away for me is that we could benefit from having each package define its own "build" command instead of handling this top-level, there are a couple advantages for this:

  • We can easily switch tools between packages avoiding lockdown (like we have right now), some tools might be better for some packages, some packages might not require building... the packages become more flexible.
  • We could start leveraging turborepo (or rush) to speedup and organize all these tasks. We could potentially increase the speed of CI jobs if we leverage remote caches as well.

The other take away for me is related to the adoption of these rust/go builders and the removal of babel: We have too many babel specific logic in our chains that we should try to eliminate over time if we want to avoid the lock-in to last forever while the JS community abandons babel.

Curious what you think about my ramblings and also the current PR (package specific builds, swc...)

@youknowriad youknowriad added the [Type] Technical Prototype Offers a technical exploration into an idea as an example of what's possible label Dec 13, 2021
@github-actions
Copy link

Size Change: +6.59 kB (+1%)

Total Size: 1.11 MB

Filename Size Change
build/a11y/index.min.js 945 B -15 B (-2%)
build/api-fetch/index.min.js 5.2 kB +2.99 kB (+135%) 🆘
build/blob/index.min.js 458 B -1 B (0%)
build/block-serialization-default-parser/index.min.js 1.59 kB +501 B (+46%) 🚨
build/dom/index.min.js 5.08 kB +586 B (+13%) ⚠️
build/element/index.min.js 4.17 kB +884 B (+27%) 🚨
build/escape-html/index.min.js 514 B -3 B (-1%)
build/hooks/index.min.js 1.7 kB +72 B (+4%)
build/i18n/index.min.js 4.16 kB +447 B (+12%) ⚠️
build/is-shallow-equal/index.min.js 494 B -7 B (-1%)
build/keycodes/index.min.js 1.81 kB +428 B (+31%) 🚨
build/priority-queue/index.min.js 578 B -4 B (-1%)
build/url/index.min.js 2.61 kB +713 B (+38%) 🚨
ℹ️ View Unchanged
Filename Size
build/admin-manifest/index.min.js 1.1 kB
build/annotations/index.min.js 2.75 kB
build/autop/index.min.js 2.12 kB
build/block-directory/index.min.js 6.28 kB
build/block-directory/style-rtl.css 1.01 kB
build/block-directory/style.css 1.01 kB
build/block-editor/default-editor-styles-rtl.css 378 B
build/block-editor/default-editor-styles.css 378 B
build/block-editor/index.min.js 140 kB
build/block-editor/style-rtl.css 14.4 kB
build/block-editor/style.css 14.4 kB
build/block-library/blocks/archives/editor-rtl.css 61 B
build/block-library/blocks/archives/editor.css 60 B
build/block-library/blocks/archives/style-rtl.css 65 B
build/block-library/blocks/archives/style.css 65 B
build/block-library/blocks/audio/editor-rtl.css 58 B
build/block-library/blocks/audio/editor.css 58 B
build/block-library/blocks/audio/style-rtl.css 111 B
build/block-library/blocks/audio/style.css 111 B
build/block-library/blocks/audio/theme-rtl.css 125 B
build/block-library/blocks/audio/theme.css 125 B
build/block-library/blocks/block/editor-rtl.css 161 B
build/block-library/blocks/block/editor.css 161 B
build/block-library/blocks/button/editor-rtl.css 470 B
build/block-library/blocks/button/editor.css 470 B
build/block-library/blocks/button/style-rtl.css 560 B
build/block-library/blocks/button/style.css 560 B
build/block-library/blocks/buttons/editor-rtl.css 291 B
build/block-library/blocks/buttons/editor.css 291 B
build/block-library/blocks/buttons/style-rtl.css 275 B
build/block-library/blocks/buttons/style.css 275 B
build/block-library/blocks/calendar/style-rtl.css 207 B
build/block-library/blocks/calendar/style.css 207 B
build/block-library/blocks/categories/editor-rtl.css 84 B
build/block-library/blocks/categories/editor.css 83 B
build/block-library/blocks/categories/style-rtl.css 79 B
build/block-library/blocks/categories/style.css 79 B
build/block-library/blocks/code/style-rtl.css 90 B
build/block-library/blocks/code/style.css 90 B
build/block-library/blocks/code/theme-rtl.css 134 B
build/block-library/blocks/code/theme.css 134 B
build/block-library/blocks/columns/editor-rtl.css 210 B
build/block-library/blocks/columns/editor.css 208 B
build/block-library/blocks/columns/style-rtl.css 503 B
build/block-library/blocks/columns/style.css 502 B
build/block-library/blocks/comment-template/style-rtl.css 127 B
build/block-library/blocks/comment-template/style.css 127 B
build/block-library/blocks/comments-pagination-numbers/editor-rtl.css 123 B
build/block-library/blocks/comments-pagination-numbers/editor.css 121 B
build/block-library/blocks/comments-pagination/editor-rtl.css 222 B
build/block-library/blocks/comments-pagination/editor.css 209 B
build/block-library/blocks/comments-pagination/style-rtl.css 235 B
build/block-library/blocks/comments-pagination/style.css 231 B
build/block-library/blocks/cover/editor-rtl.css 546 B
build/block-library/blocks/cover/editor.css 547 B
build/block-library/blocks/cover/style-rtl.css 1.22 kB
build/block-library/blocks/cover/style.css 1.22 kB
build/block-library/blocks/embed/editor-rtl.css 488 B
build/block-library/blocks/embed/editor.css 488 B
build/block-library/blocks/embed/style-rtl.css 417 B
build/block-library/blocks/embed/style.css 417 B
build/block-library/blocks/embed/theme-rtl.css 124 B
build/block-library/blocks/embed/theme.css 124 B
build/block-library/blocks/file/editor-rtl.css 300 B
build/block-library/blocks/file/editor.css 300 B
build/block-library/blocks/file/style-rtl.css 255 B
build/block-library/blocks/file/style.css 255 B
build/block-library/blocks/file/view.min.js 322 B
build/block-library/blocks/freeform/editor-rtl.css 2.44 kB
build/block-library/blocks/freeform/editor.css 2.44 kB
build/block-library/blocks/gallery/editor-rtl.css 966 B
build/block-library/blocks/gallery/editor.css 970 B
build/block-library/blocks/gallery/style-rtl.css 1.63 kB
build/block-library/blocks/gallery/style.css 1.62 kB
build/block-library/blocks/gallery/theme-rtl.css 122 B
build/block-library/blocks/gallery/theme.css 122 B
build/block-library/blocks/group/editor-rtl.css 159 B
build/block-library/blocks/group/editor.css 159 B
build/block-library/blocks/group/style-rtl.css 57 B
build/block-library/blocks/group/style.css 57 B
build/block-library/blocks/group/theme-rtl.css 78 B
build/block-library/blocks/group/theme.css 78 B
build/block-library/blocks/heading/style-rtl.css 114 B
build/block-library/blocks/heading/style.css 114 B
build/block-library/blocks/html/editor-rtl.css 332 B
build/block-library/blocks/html/editor.css 333 B
build/block-library/blocks/image/editor-rtl.css 810 B
build/block-library/blocks/image/editor.css 809 B
build/block-library/blocks/image/style-rtl.css 507 B
build/block-library/blocks/image/style.css 511 B
build/block-library/blocks/image/theme-rtl.css 124 B
build/block-library/blocks/image/theme.css 124 B
build/block-library/blocks/latest-comments/style-rtl.css 284 B
build/block-library/blocks/latest-comments/style.css 284 B
build/block-library/blocks/latest-posts/editor-rtl.css 137 B
build/block-library/blocks/latest-posts/editor.css 137 B
build/block-library/blocks/latest-posts/style-rtl.css 528 B
build/block-library/blocks/latest-posts/style.css 527 B
build/block-library/blocks/list/style-rtl.css 94 B
build/block-library/blocks/list/style.css 94 B
build/block-library/blocks/media-text/editor-rtl.css 266 B
build/block-library/blocks/media-text/editor.css 263 B
build/block-library/blocks/media-text/style-rtl.css 493 B
build/block-library/blocks/media-text/style.css 490 B
build/block-library/blocks/more/editor-rtl.css 431 B
build/block-library/blocks/more/editor.css 431 B
build/block-library/blocks/navigation-link/editor-rtl.css 649 B
build/block-library/blocks/navigation-link/editor.css 650 B
build/block-library/blocks/navigation-link/style-rtl.css 94 B
build/block-library/blocks/navigation-link/style.css 94 B
build/block-library/blocks/navigation-submenu/editor-rtl.css 299 B
build/block-library/blocks/navigation-submenu/editor.css 299 B
build/block-library/blocks/navigation-submenu/view.min.js 343 B
build/block-library/blocks/navigation/editor-rtl.css 1.91 kB
build/block-library/blocks/navigation/editor.css 1.91 kB
build/block-library/blocks/navigation/style-rtl.css 1.68 kB
build/block-library/blocks/navigation/style.css 1.67 kB
build/block-library/blocks/navigation/view.min.js 2.79 kB
build/block-library/blocks/nextpage/editor-rtl.css 395 B
build/block-library/blocks/nextpage/editor.css 395 B
build/block-library/blocks/page-list/editor-rtl.css 377 B
build/block-library/blocks/page-list/editor.css 377 B
build/block-library/blocks/page-list/style-rtl.css 172 B
build/block-library/blocks/page-list/style.css 172 B
build/block-library/blocks/paragraph/editor-rtl.css 157 B
build/block-library/blocks/paragraph/editor.css 157 B
build/block-library/blocks/paragraph/style-rtl.css 273 B
build/block-library/blocks/paragraph/style.css 273 B
build/block-library/blocks/post-author/style-rtl.css 175 B
build/block-library/blocks/post-author/style.css 176 B
build/block-library/blocks/post-comments-form/style-rtl.css 446 B
build/block-library/blocks/post-comments-form/style.css 446 B
build/block-library/blocks/post-comments/style-rtl.css 507 B
build/block-library/blocks/post-comments/style.css 507 B
build/block-library/blocks/post-excerpt/editor-rtl.css 73 B
build/block-library/blocks/post-excerpt/editor.css 73 B
build/block-library/blocks/post-excerpt/style-rtl.css 69 B
build/block-library/blocks/post-excerpt/style.css 69 B
build/block-library/blocks/post-featured-image/editor-rtl.css 721 B
build/block-library/blocks/post-featured-image/editor.css 721 B
build/block-library/blocks/post-featured-image/style-rtl.css 153 B
build/block-library/blocks/post-featured-image/style.css 153 B
build/block-library/blocks/post-template/editor-rtl.css 99 B
build/block-library/blocks/post-template/editor.css 98 B
build/block-library/blocks/post-template/style-rtl.css 391 B
build/block-library/blocks/post-template/style.css 392 B
build/block-library/blocks/post-terms/style-rtl.css 73 B
build/block-library/blocks/post-terms/style.css 73 B
build/block-library/blocks/post-title/style-rtl.css 80 B
build/block-library/blocks/post-title/style.css 80 B
build/block-library/blocks/preformatted/style-rtl.css 103 B
build/block-library/blocks/preformatted/style.css 103 B
build/block-library/blocks/pullquote/editor-rtl.css 198 B
build/block-library/blocks/pullquote/editor.css 198 B
build/block-library/blocks/pullquote/style-rtl.css 389 B
build/block-library/blocks/pullquote/style.css 388 B
build/block-library/blocks/pullquote/theme-rtl.css 167 B
build/block-library/blocks/pullquote/theme.css 167 B
build/block-library/blocks/query-pagination-numbers/editor-rtl.css 122 B
build/block-library/blocks/query-pagination-numbers/editor.css 121 B
build/block-library/blocks/query-pagination/editor-rtl.css 221 B
build/block-library/blocks/query-pagination/editor.css 211 B
build/block-library/blocks/query-pagination/style-rtl.css 273 B
build/block-library/blocks/query-pagination/style.css 269 B
build/block-library/blocks/query/editor-rtl.css 131 B
build/block-library/blocks/query/editor.css 132 B
build/block-library/blocks/quote/style-rtl.css 187 B
build/block-library/blocks/quote/style.css 187 B
build/block-library/blocks/quote/theme-rtl.css 223 B
build/block-library/blocks/quote/theme.css 226 B
build/block-library/blocks/rss/editor-rtl.css 202 B
build/block-library/blocks/rss/editor.css 204 B
build/block-library/blocks/rss/style-rtl.css 289 B
build/block-library/blocks/rss/style.css 288 B
build/block-library/blocks/search/editor-rtl.css 165 B
build/block-library/blocks/search/editor.css 165 B
build/block-library/blocks/search/style-rtl.css 397 B
build/block-library/blocks/search/style.css 398 B
build/block-library/blocks/search/theme-rtl.css 64 B
build/block-library/blocks/search/theme.css 64 B
build/block-library/blocks/separator/editor-rtl.css 99 B
build/block-library/blocks/separator/editor.css 99 B
build/block-library/blocks/separator/style-rtl.css 245 B
build/block-library/blocks/separator/style.css 245 B
build/block-library/blocks/separator/theme-rtl.css 172 B
build/block-library/blocks/separator/theme.css 172 B
build/block-library/blocks/shortcode/editor-rtl.css 474 B
build/block-library/blocks/shortcode/editor.css 474 B
build/block-library/blocks/site-logo/editor-rtl.css 772 B
build/block-library/blocks/site-logo/editor.css 772 B
build/block-library/blocks/site-logo/style-rtl.css 165 B
build/block-library/blocks/site-logo/style.css 165 B
build/block-library/blocks/site-tagline/editor-rtl.css 86 B
build/block-library/blocks/site-tagline/editor.css 86 B
build/block-library/blocks/site-title/editor-rtl.css 84 B
build/block-library/blocks/site-title/editor.css 84 B
build/block-library/blocks/social-link/editor-rtl.css 177 B
build/block-library/blocks/social-link/editor.css 177 B
build/block-library/blocks/social-links/editor-rtl.css 670 B
build/block-library/blocks/social-links/editor.css 669 B
build/block-library/blocks/social-links/style-rtl.css 1.32 kB
build/block-library/blocks/social-links/style.css 1.32 kB
build/block-library/blocks/spacer/editor-rtl.css 307 B
build/block-library/blocks/spacer/editor.css 307 B
build/block-library/blocks/spacer/style-rtl.css 48 B
build/block-library/blocks/spacer/style.css 48 B
build/block-library/blocks/table/editor-rtl.css 471 B
build/block-library/blocks/table/editor.css 472 B
build/block-library/blocks/table/style-rtl.css 481 B
build/block-library/blocks/table/style.css 481 B
build/block-library/blocks/table/theme-rtl.css 188 B
build/block-library/blocks/table/theme.css 188 B
build/block-library/blocks/tag-cloud/style-rtl.css 146 B
build/block-library/blocks/tag-cloud/style.css 146 B
build/block-library/blocks/template-part/editor-rtl.css 560 B
build/block-library/blocks/template-part/editor.css 559 B
build/block-library/blocks/template-part/theme-rtl.css 101 B
build/block-library/blocks/template-part/theme.css 101 B
build/block-library/blocks/text-columns/editor-rtl.css 95 B
build/block-library/blocks/text-columns/editor.css 95 B
build/block-library/blocks/text-columns/style-rtl.css 166 B
build/block-library/blocks/text-columns/style.css 166 B
build/block-library/blocks/verse/style-rtl.css 87 B
build/block-library/blocks/verse/style.css 87 B
build/block-library/blocks/video/editor-rtl.css 569 B
build/block-library/blocks/video/editor.css 570 B
build/block-library/blocks/video/style-rtl.css 173 B
build/block-library/blocks/video/style.css 173 B
build/block-library/blocks/video/theme-rtl.css 124 B
build/block-library/blocks/video/theme.css 124 B
build/block-library/common-rtl.css 857 B
build/block-library/common.css 856 B
build/block-library/editor-rtl.css 10 kB
build/block-library/editor.css 10 kB
build/block-library/index.min.js 163 kB
build/block-library/reset-rtl.css 474 B
build/block-library/reset.css 474 B
build/block-library/style-rtl.css 10.7 kB
build/block-library/style.css 10.8 kB
build/block-library/theme-rtl.css 672 B
build/block-library/theme.css 677 B
build/block-serialization-spec-parser/index.min.js 2.79 kB
build/blocks/index.min.js 46.3 kB
build/components/index.min.js 215 kB
build/components/style-rtl.css 15.5 kB
build/components/style.css 15.5 kB
build/compose/index.min.js 10.9 kB
build/core-data/index.min.js 13.2 kB
build/customize-widgets/index.min.js 11.4 kB
build/customize-widgets/style-rtl.css 1.5 kB
build/customize-widgets/style.css 1.49 kB
build/data-controls/index.min.js 631 B
build/data/index.min.js 7.49 kB
build/date/index.min.js 31.5 kB
build/deprecated/index.min.js 485 B
build/dom-ready/index.min.js 304 B
build/edit-navigation/index.min.js 16 kB
build/edit-navigation/style-rtl.css 3.76 kB
build/edit-navigation/style.css 3.76 kB
build/edit-post/classic-rtl.css 492 B
build/edit-post/classic.css 494 B
build/edit-post/index.min.js 29.3 kB
build/edit-post/style-rtl.css 7.1 kB
build/edit-post/style.css 7.09 kB
build/edit-site/index.min.js 35.4 kB
build/edit-site/style-rtl.css 6.57 kB
build/edit-site/style.css 6.57 kB
build/edit-widgets/index.min.js 16.5 kB
build/edit-widgets/style-rtl.css 4.18 kB
build/edit-widgets/style.css 4.18 kB
build/editor/index.min.js 37.8 kB
build/editor/style-rtl.css 3.78 kB
build/editor/style.css 3.77 kB
build/format-library/index.min.js 6.58 kB
build/format-library/style-rtl.css 571 B
build/format-library/style.css 571 B
build/html-entities/index.min.js 424 B
build/keyboard-shortcuts/index.min.js 1.8 kB
build/list-reusable-blocks/index.min.js 1.72 kB
build/list-reusable-blocks/style-rtl.css 838 B
build/list-reusable-blocks/style.css 838 B
build/media-utils/index.min.js 2.92 kB
build/notices/index.min.js 925 B
build/nux/index.min.js 2.08 kB
build/nux/style-rtl.css 747 B
build/nux/style.css 743 B
build/plugins/index.min.js 1.84 kB
build/primitives/index.min.js 924 B
build/react-i18n/index.min.js 671 B
build/redux-routine/index.min.js 2.65 kB
build/reusable-blocks/index.min.js 2.22 kB
build/reusable-blocks/style-rtl.css 256 B
build/reusable-blocks/style.css 256 B
build/rich-text/index.min.js 11 kB
build/server-side-render/index.min.js 1.57 kB
build/shortcode/index.min.js 1.49 kB
build/token-list/index.min.js 639 B
build/viewport/index.min.js 1.05 kB
build/warning/index.min.js 248 B
build/widgets/index.min.js 7.15 kB
build/widgets/style-rtl.css 1.16 kB
build/widgets/style.css 1.16 kB
build/wordcount/index.min.js 1.04 kB

compressed-size-action

@gziolo gziolo added the [Type] Build Tooling Issues or PRs related to build tooling label Dec 13, 2021
@gziolo gziolo changed the title Update/faster builds Build Tooling: Explore faster builds split into individual packages Dec 13, 2021
@youknowriad youknowriad mentioned this pull request Dec 13, 2021
Copy link
Member

@jsnajdr jsnajdr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The swc migration is surprisingly straightforward! It helps that Gutenberg doesn't use any custom plugins. Except @emotion/babel-plugin which optimizes the css syntax. That will need to be addressed somehow.

I liked the centralized bin/packages/build.js script. I don't see much upside to copying the same stuff over and over in each of the package.jsons.

@@ -28,11 +28,25 @@
"react-native": "src/index",
"types": "build-types",
"dependencies": {
"@babel/runtime": "^7.16.0",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The @babel/runtime will need to be replaced by @swc/helpers and the jsc.externalHelpers = true config option. Otherwise all the helpers (for more complex transpilations like classes, async, iterators, ...) would be duplicated in each compiled file.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I saw that, I wanted to check how much of a bundle size impact it would have to avoid the helpers.

@jsnajdr
Copy link
Member

jsnajdr commented Dec 13, 2021

@youknowriad Does this patch already compile JSX to @wordpress/element? Seems to me it's not configured anywhere and defaults to React.

To make it work with SWC, we'll probably need to move to the new automatic runtime. With the classic runtime, SWC doesn't have the equivalent of Babel's source option:

[
  require.resolve( '@wordpress/babel-plugin-import-jsx-pragma' ),
  {
    scopeVariable: 'createElement',
    scopeVariableFrag: 'Fragment',
    source: '@wordpress/element',
     isDefault: false,
  },
]

SWC lets you specify the jsc.transform.react.pragma{Frag} options, but they refer to global variables and can't be matched automatically to a certain import statement.

With runtime: automatic, setting jsc.transform.react.importSource = '@wordpress/element/jsx-runtime' will work once something like #34221 gets landed.

@youknowriad
Copy link
Contributor Author

@jsnajdr it doesn't compile JSX yet, and the reason is mentioned on the description. (you're right about move to automatic but it's not that simple)

@gziolo
Copy link
Member

gziolo commented Dec 13, 2021

The swc migration is surprisingly straightforward! It helps that Gutenberg doesn't use any custom plugins. Except @emotion/babel-plugin which optimizes the css syntax. That will need to be addressed somehow.

This is on the Next.js roadmap as noted in https://nextjs.org/blog/next-12:

The new Rust compiler is backwards compatible. If you have an existing Babel configuration, you will automatically be opted out. We have plans to port parsing for popular libraries like styled-components, emotion, and relay soon. If you're using a custom Babel setup, please share your configuration.

With runtime: automatic, setting jsc.transform.react.importSource = '@wordpress/element/jsx-runtime' will work once something like #34221 gets landed.

While this would work great for the Gutenberg plugin, it is suboptimal for npm packages published to npm. As far as I'm aware, libraries still ship code that uses direct access to createElement from react. JSX runtime changes dynamically depending on whether you use production or development mode, but it also has special handling when profiling your React app.

@gziolo
Copy link
Member

gziolo commented Dec 13, 2021

There is also an open discussion started by @kevin940726 in #36142 about the general improvements to the build development workflow where we could summarize all the findings from this PR and from the efforts to start using pnpm.

@mtias mtias added the [Type] Code Quality Issues or PRs that relate to code quality label Dec 15, 2021
@youknowriad youknowriad deleted the update/faster-builds branch September 19, 2022 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[Type] Build Tooling Issues or PRs related to build tooling [Type] Code Quality Issues or PRs that relate to code quality [Type] Technical Prototype Offers a technical exploration into an idea as an example of what's possible
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants