Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consume monorepo packages without building them #43679

Draft
wants to merge 11 commits into
base: trunk
Choose a base branch
from
Draft

Conversation

jsnajdr
Copy link
Member

@jsnajdr jsnajdr commented Jun 25, 2020

This PR makes it possible to consume monorepo packages (packages/*) without building them. To build Calypso, i.e., its webpack bundle, the dist/esm and dist/cjs folders don't need to be created and populated with transpiled files. That's a task that's needed only when preparing a package to be published in the NPM registry.

Instead, the Calypso webpack build can now directly consume the sources in src/, and transpile any Babel-enhanced JavaScript or TypeScript with its webpack loaders, just like any other non-package file in client/.

I achieved this by adding calypso:main field to all packages, and pointing in to the ./src/index source file. Then I configured the webpack resolver to look at calypso:main first, using the resolve.mainFields option.

To make tests work, the Jest resolver needs to be aware of calypso:main, too. That's harder, and I had to write a custom resolver (packages/calypso-build/jest/calypso-resolver) by copying and modifying the default resolver from Jest sources.

Building Calypso, both in development (with the dev middleware that watches the sources for changes) and production, now doesn't require the build-packages or build-packages:watch steps. Simply running webpack is enough.

The only package that still needs the build step is calypso-color-schemes. It does SCSS-to-CSS-and-JS compilation, and also performs some magic with copying color-schemes assets (I remember that @ockham had to add that at some point to solve some problem that didn't have any other solution)

Risks:
There are several independent implementations of the module resolution algorithm in various tools:

  • webpack
  • TypeScript
  • Jest
  • Node.js

And not all of them can be easily configured to look at the calypso:main fields. We might end up with some part of our build pipeline being unable to load the monorepo packages correctly.

How to test:
Run yarn start and see if it builds Calypso correctly and if the build is faster now.

@jsnajdr jsnajdr added [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. Build labels Jun 25, 2020
@jsnajdr jsnajdr requested review from ockham, sgomes, sirreal, alshakero and a team June 25, 2020 12:07
@jsnajdr jsnajdr self-assigned this Jun 25, 2020
@matticbot
Copy link
Contributor

@matticbot
Copy link
Contributor

Caution: This PR affects files in the FSE Plugin on WordPress.com
Please ensure your changes work on WordPress.com before merging.

D45477-code has been created so you can easily test it on your sandbox. See this FieldGuide page about developing in the FSE Plugin for more info: PCYsg-ly5-p2

@jsnajdr jsnajdr force-pushed the build/no-packages branch from e7fabb6 to 3bf05d8 Compare June 25, 2020 12:19
@matticbot
Copy link
Contributor

matticbot commented Jun 25, 2020

Here is how your PR affects size of JS and CSS bundles shipped to the user's browser:

Webpack Runtime (~193 bytes removed 📉 [gzipped])

name      parsed_size           gzip_size
manifest       -801 B  (-0.4%)     -193 B  (-0.5%)

Webpack runtime for loading modules. It is included in the HTML page as an inline script. Is downloaded and parsed every time the app is loaded.

App Entrypoints (~10235 bytes removed 📉 [gzipped])

name                   parsed_size           gzip_size
entry-main                -22642 B  (-1.6%)    -3136 B  (-0.9%)
entry-login               -19352 B  (-2.2%)    -3473 B  (-1.4%)
entry-domains-landing     -15352 B  (-2.7%)    -1805 B  (-1.2%)
entry-gutenboarding        -9203 B  (-0.5%)    -1821 B  (-0.4%)

Common code that is always downloaded and parsed every time the app is loaded, no matter which route is used.

Sections (~24051 bytes removed 📉 [gzipped])

name                      parsed_size           gzip_size
checkout                     -15456 B  (-1.0%)    -3770 B  (-1.0%)
domains                       -4524 B  (-0.4%)    -1605 B  (-0.6%)
jetpack-connect               -4016 B  (-0.6%)    -1921 B  (-1.1%)
zoninator                     -2897 B  (-1.0%)    -1059 B  (-1.2%)
plans                         -1777 B  (-0.4%)     -582 B  (-0.4%)
woocommerce                   -1768 B  (-0.1%)     -517 B  (-0.1%)
stats                         -1766 B  (-0.2%)     -543 B  (-0.3%)
activity                      -1619 B  (-0.3%)     -515 B  (-0.4%)
media                         -1618 B  (-0.4%)     -495 B  (-0.5%)
themes                        -1615 B  (-0.5%)     -519 B  (-0.5%)
settings                      -1600 B  (-0.4%)     -356 B  (-0.3%)
plugins                       -1552 B  (-0.4%)     -497 B  (-0.5%)
gutenberg-editor              +1532 B  (+0.2%)     +138 B  (+0.1%)
backup                        -1531 B  (-0.3%)     -482 B  (-0.4%)
settings-performance          -1494 B  (-0.6%)     -519 B  (-0.7%)
import                        -1419 B  (-0.7%)     -289 B  (-0.5%)
post-editor                   -1400 B  (-0.1%)     -231 B  (-0.0%)
purchases                     -1329 B  (-0.1%)     -322 B  (-0.1%)
security                      -1029 B  (-0.2%)     -296 B  (-0.2%)
reader                        -1002 B  (-0.2%)     -301 B  (-0.2%)
marketing                      -953 B  (-0.2%)     -302 B  (-0.3%)
help                           -943 B  (-0.2%)     -304 B  (-0.3%)
settings-writing               -931 B  (-0.2%)     -303 B  (-0.2%)
migrate                        -925 B  (-0.7%)     -282 B  (-0.8%)
theme                          -919 B  (-0.3%)     -271 B  (-0.4%)
accept-invite                  -919 B  (-0.3%)     -990 B  (-1.2%)
email                          -913 B  (-0.3%)     -278 B  (-0.3%)
account                        -898 B  (-0.2%)     -287 B  (-0.3%)
scan                           -892 B  (-0.3%)     -273 B  (-0.3%)
account-close                  -883 B  (-0.3%)     -284 B  (-0.3%)
posts                          -882 B  (-0.3%)     -305 B  (-0.3%)
settings-security              -878 B  (-0.3%)     -297 B  (-0.4%)
earn                           -877 B  (-0.3%)     -269 B  (-0.4%)
posts-custom                   -874 B  (-0.3%)     -302 B  (-0.3%)
notification-settings          -869 B  (-0.2%)     -285 B  (-0.3%)
google-my-business             -865 B  (-0.3%)     -219 B  (-0.3%)
comments                       -852 B  (-0.2%)     -283 B  (-0.2%)
site-blocks                    -846 B  (-0.3%)     -281 B  (-0.3%)
privacy                        -841 B  (-0.3%)     -278 B  (-0.4%)
happychat                      -837 B  (-0.3%)     -280 B  (-0.4%)
people                         -835 B  (-0.2%)     -280 B  (-0.3%)
settings-jetpack               -833 B  (-0.4%)     -226 B  (-0.4%)
home                           -829 B  (-0.2%)     -270 B  (-0.3%)
me                             -828 B  (-0.3%)     -274 B  (-0.4%)
hosting                        -828 B  (-0.4%)     -254 B  (-0.4%)
wp-super-cache                 -801 B  (-0.4%)     -280 B  (-0.5%)
settings-discussion            -796 B  (-0.4%)     -285 B  (-0.5%)
export                         -789 B  (-0.4%)     -280 B  (-0.5%)
pages                          -779 B  (-0.3%)     -270 B  (-0.4%)
hello-dolly                    -737 B  (-0.7%)     -189 B  (-0.6%)
sensei                         -723 B  (-0.8%)     -178 B  (-0.6%)
signup                         -357 B  (-0.2%)      -26 B  (-0.0%)
concierge                      -239 B  (-0.1%)      -29 B  (-0.0%)
jetpack-cloud-settings         -161 B  (-0.1%)      -29 B  (-0.1%)
customize                      -122 B  (-0.1%)      -34 B  (-0.1%)
feature-upsell                 -120 B  (-0.1%)      -29 B  (-0.1%)
preview                        -116 B  (-0.1%)      -31 B  (-0.1%)
sites                           -97 B  (-0.1%)      -22 B  (-0.1%)
devdocs                         -64 B  (-0.0%)       -5 B  (-0.0%)
domain-connect-authorize        -30 B  (-0.2%)       +1 B  (+0.0%)
auth                            -28 B  (-0.1%)       -7 B  (-0.1%)

Sections contain code specific for a given set of routes. Is downloaded and parsed only when a particular route is navigated to.

Async-loaded Components (~15528 bytes removed 📉 [gzipped])

name                                                                  parsed_size            gzip_size
async-load-design-playground                                             -12506 B   (-0.7%)    -3041 B   (-0.7%)
async-load-design                                                        -12343 B   (-0.7%)    -2832 B   (-0.6%)
async-load-my-sites-current-site-notice                                   -9383 B  (-17.2%)    -1705 B  (-12.2%)
async-load-design-blocks                                                  -4848 B   (-0.2%)    -1082 B   (-0.2%)
async-load-signup-steps-about                                             -3307 B   (-5.7%)    -1138 B   (-6.9%)
async-load-signup-steps-site-topic                                        -2667 B   (-9.1%)     -898 B  (-10.7%)
async-load-signup-steps-domains                                           -1452 B   (-0.6%)     -478 B   (-0.8%)
async-load-signup-steps-plans                                              -927 B   (-0.5%)     -269 B   (-0.6%)
async-load-components-web-preview-component                                -900 B   (-0.2%)     -289 B   (-0.3%)
async-load-post-editor-media-modal                                         -886 B   (-0.3%)     -251 B   (-0.3%)
async-load-signup-steps-user                                               -875 B   (-0.7%)     -980 B   (-2.9%)
async-load-signup-steps-plans-atomic-store                                 -856 B   (-0.8%)     -256 B   (-0.9%)
async-load-signup-steps-clone-point                                        -753 B   (-0.5%)     -223 B   (-0.6%)
async-load-signup-steps-theme-selection                                    -712 B   (-1.1%)     -238 B   (-1.3%)
async-load-my-sites-sidebar                                                -701 B   (-0.4%)     -262 B   (-0.6%)
async-load-signup-steps-clone-credentials                                  -695 B   (-1.1%)     -223 B   (-1.3%)
async-load-components-sites-popover                                        -664 B   (-1.1%)     -243 B   (-1.4%)
async-load-layout-guided-tours-component                                   -649 B   (-0.7%)     -234 B   (-1.0%)
async-load-layout-masterbar-drafts-popover                                 -631 B   (-1.9%)     -206 B   (-2.1%)
async-load-blocks-inline-help                                              -628 B   (-1.0%)     -223 B   (-1.5%)
async-load-blocks-inline-help-popover                                      -197 B   (-0.1%)      -57 B   (-0.1%)
async-load-signup-steps-import-url-onboarding                               -87 B   (-0.2%)      -16 B   (-0.1%)
async-load-signup-steps-site-picker                                         -77 B   (-0.1%)      -19 B   (-0.1%)
async-load-extensions-woocommerce-app-store-stats-referrers                 -76 B   (-0.2%)      -11 B   (-0.1%)
async-load-signup-steps-import-url                                          -73 B   (-0.3%)      -11 B   (-0.2%)
async-load-extensions-woocommerce-app-store-stats                           -67 B   (-0.2%)       -8 B   (-0.1%)
async-load-reader-following-manage                                          -64 B   (-0.1%)       -8 B   (-0.0%)
async-load-blocks-support-article-dialog-dialog                             -64 B   (-0.1%)      -16 B   (-0.1%)
async-load-signup-steps-import-preview                                      -63 B   (-0.2%)      -10 B   (-0.1%)
async-load-signup-steps-clone-destination                                   -58 B   (-0.3%)       -8 B   (-0.2%)
async-load-reader-sidebar                                                   -52 B   (-0.1%)       -1 B   (-0.0%)
async-load-my-sites-guided-transfer                                         -49 B   (-0.2%)       -4 B   (-0.1%)
async-load-my-sites-current-site-domain-warnings                            -48 B   (-0.1%)      -11 B   (-0.1%)
async-load-signup-steps-upsell                                              -45 B   (-0.3%)       +2 B   (+0.0%)
async-load-blocks-legal-updates-banner                                      -44 B   (-0.9%)       -8 B   (-0.5%)
async-load-layout-nps-survey-notice                                         -43 B   (-0.2%)      -10 B   (-0.2%)
async-load-extensions-woocommerce-app-store-stats-listview                  -43 B   (-0.3%)       -7 B   (-0.2%)
async-load-signup-steps-rewind-form-creds                                   -42 B   (-0.1%)      -10 B   (-0.1%)
async-load-reader-site-stream                                               -42 B   (-0.2%)       -9 B   (-0.1%)
async-load-reader-feed-stream                                               -42 B   (-0.2%)       -9 B   (-0.1%)
async-load-blocks-product-purchase-features-list                            -41 B   (-0.1%)       -3 B   (-0.0%)
async-load-reader-search-stream                                             -37 B   (-0.0%)       -6 B   (-0.0%)
async-load-reader-list-manage                                               -37 B   (-0.1%)       +3 B   (+0.0%)
async-load-components-jetpack-sidebar                                       -37 B   (-0.2%)       -7 B   (-0.1%)
async-load-signup-steps-site                                                -36 B   (-0.1%)       -9 B   (-0.1%)
async-load-quick-language-switcher                                          -34 B   (-0.1%)       -9 B   (-0.1%)
async-load-blocks-reader-full-post                                          -33 B   (-0.1%)       -8 B   (-0.1%)
async-load-signup-steps-passwordless                                        -31 B   (-0.1%)       -7 B   (-0.1%)
async-load-post-editor-editor-sharing-accordion                             -29 B   (-0.1%)       -3 B   (-0.0%)
async-load-signup-steps-p2-site                                             -28 B   (-0.1%)       -8 B   (-0.1%)
async-load-reader-tag-stream-main                                           -28 B   (-0.2%)       -5 B   (-0.1%)
async-load-signup-steps-survey                                              -27 B   (-0.2%)       -5 B   (-0.1%)
async-load-signup-steps-rewind-migrate                                      -27 B   (-0.1%)       -8 B   (-0.1%)
async-load-signup-steps-creds-permission                                    -27 B   (-0.1%)       -4 B   (-0.0%)
async-load-signup-steps-creds-confirm                                       -27 B   (-0.1%)       -5 B   (-0.1%)
async-load-signup-steps-clone-ready                                         -27 B   (-0.1%)       -8 B   (-0.1%)
async-load-layout-guided-tours                                              -25 B   (-0.2%)       -5 B   (-0.1%)
async-load-signup-steps-site-style                                          -23 B   (-0.1%)       -5 B   (-0.1%)
async-load-signup-steps-rebrand-cities-welcome                              -23 B   (-0.2%)       -1 B   (-0.0%)
async-load-layout-community-translator-launcher                             -23 B   (-0.1%)       -4 B   (-0.1%)
async-load-blocks-login-two-factor-authentication-two-factor-content        -23 B   (-0.1%)       -4 B   (-0.1%)
async-load-signup-steps-site-type                                           -22 B   (-0.2%)       -3 B   (-0.1%)
async-load-signup-steps-site-or-domain                                      -22 B   (-0.1%)       -6 B   (-0.1%)
async-load-signup-steps-rewind-were-backing                                 -22 B   (-0.3%)       -3 B   (-0.1%)
async-load-signup-steps-reader-landing                                      -22 B   (-0.2%)       -5 B   (-0.2%)
async-load-signup-steps-creds-complete                                      -22 B   (-0.3%)       -5 B   (-0.2%)
async-load-signup-steps-clone-start                                         -22 B   (-0.2%)       -3 B   (-0.1%)
async-load-signup-steps-clone-jetpack                                       -22 B   (-0.2%)       -5 B   (-0.2%)
async-load-signup-steps-clone-cloning                                       -22 B   (-0.2%)       -6 B   (-0.1%)
async-load-blocks-jitm-templates-sidebar-banner                             -22 B   (-0.1%)       -4 B   (-0.1%)
async-load-blocks-jitm-templates-notice                                     -22 B   (-0.1%)       -4 B   (-0.1%)
async-load-blocks-jitm-templates-default                                    -22 B   (-0.1%)       -4 B   (-0.1%)
async-load-my-sites-current-site-stale-cart-items-notice                    -20 B   (-0.0%)      -11 B   (-0.1%)
async-load-blocks-calendar-popover                                          -20 B   (-0.0%)       -5 B   (-0.0%)
async-load-signup-steps-site-title                                          -18 B   (-0.2%)       -4 B   (-0.1%)
async-load-my-sites-site-settings-seo-settings-form                         -18 B   (-0.0%)       -2 B   (-0.0%)
async-load-blocks-login-social-connect-prompt                               -18 B   (-0.4%)       -4 B   (-0.3%)
async-load-reader-list-stream                                               -14 B   (-0.1%)       -4 B   (-0.1%)
async-load-post-editor-editor-seo-accordion                                 -14 B   (-0.2%)       -6 B   (-0.2%)
async-load-post-editor-editor-location                                      -14 B   (-0.1%)       -4 B   (-0.1%)
async-load-signup-steps-test-step                                           -13 B   (-0.2%)       -4 B   (-0.2%)
async-load-components-community-translator                                  -13 B   (-0.2%)       -8 B   (-0.3%)
async-load-blocks-inline-help-dialog                                        -13 B   (-0.3%)       -6 B   (-0.3%)
async-load-blocks-app-banner                                                -13 B   (-0.1%)       -9 B   (-0.3%)

React components that are loaded lazily, when a certain part of UI is displayed for the first time.

Legend

What is parsed and gzip size?

Parsed Size: Uncompressed size of the JS and CSS files. This much code needs to be parsed and stored in memory.
Gzip Size: Compressed size of the JS and CSS files. This much data needs to be downloaded over network.

Generated by performance advisor bot at iscalypsofastyet.com.

Copy link
Collaborator

@wp-desktop wp-desktop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WordPress Desktop CI Failure (ci/wp-desktop): @jsnajdr please re-try this workflow ("Rerun Workflow from Failed") and/or review this PR for breaking changes. Please also ensure this branch is rebased off latest Calypso master.

@jsnajdr jsnajdr requested a review from sirbrillig June 25, 2020 12:46
@nsakaimbo
Copy link
Contributor

Haven't seen this one before:

Application entry file "/Users/distiller/wp-desktop/release/mac/WordPress.com.app/Contents/Resources/app/build/desktop.js" does not exist. Seems like a wrong configuration.  stackTrace=

Copy link
Member

@alshakero alshakero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you so much for this Jarda!

I reviewed this and it seems to build correctly and works and is faster!

I investigated the caveats you mentioned a bit and I think I've come up with a solution for TypeScript.

TypeScript

Currently, TypeScript fails because types field in package.json (of a given package) points to a possibly non-existing file. Changing types field to the source file (src/index.ts) fixes the issue but is bad because if the package is ever published, src theoretically won't be published with it.

An OK solution is to add an index.ts file in the root of every TS pacakge. It should only be:

export * from './src/index';

💡 And npmignore this file.

This works because TS mimics the Node resolving algorithm, an index.ts file will tip it off to find the types file (in src) before failing over to types field in package.json. And when the package is published, index.ts won't exist, and types field will take precedence and the types in dist will be used.

Maybe we can create a nice comment and add it inside this index.ts file to explain its existence.

clean:packages

Currently, there is a still a package.json script called "clean:packages". It runs when node_modules is outdated and it takes some time. I think this can go 🎉

calypso-color-schemes

The only package that still needs the build step is calypso-color-schemes

I think a fair solution is to rename its output to something other than dist and remove the output files from gitignore. The output is CSS (or the equivalent), I don't think it will be OS or Node version dependant. No need to rebuild them on everyone's machine.

If you'd like me to execute these changes I'd be happy to.

@sgomes
Copy link
Contributor

sgomes commented Jun 30, 2020

@jsnajdr: the bundle size savings in this PR are interesting. My guess is that they come from packages now being compiled with evergreen build settings in the evergreen build (which is the one ICFY measures)?

@jsnajdr
Copy link
Member Author

jsnajdr commented Jun 30, 2020

My guess is that they come from packages now being compiled with evergreen build settings in the evergreen build (which is the one ICFY measures)?

Yes, I didn't investigate it in detail, but I think the savings come from two places:

  • transpiling the package source with BROWSERSLIST_ENV=evergreen rather than with defaults
  • the monorepo packages are no longer placed in the vendors cache group, making the chunk layout more optimal

I'm not sure if the second thing (no vendors) is already happening, or if we need to nudge webpack to actually do it. It looks like a good thing to do: only third party libraries like React should be in vendors chunks.

@jsnajdr
Copy link
Member Author

jsnajdr commented Jun 30, 2020

an index.ts file will tip it off to find the types file (in src) before failing over to types field in package.json.

That would require the resolver to first look for package-dir/index.ts before looking at package-dir/package.json and finding the types field there. But the resolver does the lookup in the opposite order.

But this package-dir/index.ts is used in case the dist/* directories don't exist, and the lookup by package.json fields (main, module, types) fails to find the target file. Then package-dir/index.ts is used as a fallback.

If the dist/ directory exists, then the files from there will be used and index.ts will be ignored. That's OK.

After implementing this though, I have a little problem with the patch-types.ts modules:

client/landing/gutenboarding/gutenberg-types-patch.ts:2:7 - error TS2300: Duplicate identifier 'breakpoint'.

2  type breakpoint = 'huge' | 'wide' | 'large' | 'medium' | 'small' | 'mobile';
        ~~~~~~~~~~

  packages/plans-grid/src/types-patch.ts:2:7
    2  type breakpoint = 'huge' | 'wide' | 'large' | 'medium' | 'small' | 'mobile';
            ~~~~~~~~~~
    'breakpoint' was also declared here.

client/landing/gutenboarding/gutenberg-types-patch.ts:3:7 - error TS2300: Duplicate identifier 'operator'.

3  type operator = '>=' | '<';
        ~~~~~~~~

  packages/plans-grid/src/types-patch.ts:3:7
    3  type operator = '>=' | '<';
            ~~~~~~~~
    'operator' was also declared here.

packages/plans-grid/src/types-patch.ts:1:1 - error TS1208: All files must be modules when the '--isolatedModules' flag is provided.

1 declare module '@wordpress/compose' {
  ~~~~~~~

Apparently, if the tsc build uses the *.d.ts files from dist/types, then the packages/plans-grid/dist/types/types-patch.d.ts is never loaded. But from src/, the corresponding source file is loaded and triggers a duplication error.

@alshakero @razvanpapadopol I assume that the "types patch" is there to monkey-patch a bug in the @wordpress/compose package typings? If so, I think it would help to deduplicate them by splitting a new, common package.

@alshakero
Copy link
Member

alshakero commented Jun 30, 2020

But the resolver does the lookup in the opposite order.

TIL. I added index.ts and it worked, I assumed it looks there before checking types field.

@alshakero @razvanpapadopol I assume that the "types patch" is there to monkey-patch a bug in the @wordpress/compose package typings?

That's right.

If so, I think it would help to deduplicate them by splitting a new, common package.

Sounds like a good solution

@jsnajdr
Copy link
Member Author

jsnajdr commented Jun 30, 2020

The typecheck-strict and transpile tasks continue to be a challenge, because they directly use the tsc executable.

If you load the TypeScript compiler programmatically from the typescript package and invoke it, you can pass a "language services host" object that implements a custom resolveModuleNames method. That's what webpack ts-loader does, for example.

But configuring the tsc executable is not possible. The TypeScript repo has had an issue open since 2017: microsoft/TypeScript#18896

What's kind of a good news is that the Yarn PnP project needs to do the same thing as we do here: pass a custom resolver to various tools. We're not alone. Yarn PnP works with webpack, ts-loader, with Jest. The only gap is the tsc compiler itself. More info is on this Yarn docs page: https://classic.yarnpkg.com/en/docs/pnp/getting-started/

Maybe we could stop using tsc directly, in favor of a wrapper script? This comment from the TS team lead suggest it's normal and expected 🙂

Note that this is already an option for non-tsc environments because the host object given to the compiler exposes resolveModuleNames which hosts may implement to their liking.

I'm not sure there's anyone who would want this ability who isn't already loading TS through some more advanced build pipeline.

@jsnajdr jsnajdr force-pushed the build/no-packages branch from ebee6f5 to d77465d Compare June 30, 2020 13:38
@scinos
Copy link
Contributor

scinos commented Aug 12, 2020

A few notes:

  • Writing our custom resolver is very tricky, as it won't support clearing the cache when Jest is run in watch mode. Unfortunately Jest doesn't provide a better way right now. I raised a PR with them to allow reusing the defaultResolver (so we get cache management for free) and be able to use packageFilter to look for other fields. More info in Support the "module" package.json field jestjs/jest#2702 (comment)

  • The change in this PR is too big in scope. I think we could ignore TypeScript packages for now and focus on pure JavaScript packages which should be easier. It will already be a big improvement (~38% faster installations according to my local tests and smaller bundle sizes for evergreen).

  • I'm not sold on the idea of introducing calypso:main, it will be yet another non standard "calypsoism" that people will have to understand and maintain. I think we could change the field module to point to the untranspiled source code (already in ESM), and leave main with the CJS transpilation for published packages.

@scinos
Copy link
Contributor

scinos commented Aug 17, 2020

Raised #44824 to cover some of the intent of this PR (do not build JS packages, leave TS packages for now)

@jsnajdr
Copy link
Member Author

jsnajdr commented Aug 17, 2020

Writing our custom resolver is very tricky, as it won't support clearing the cache when Jest is run in watch mode.

If I understand it correctly, that's now solved with your Jest PR and your Jest resolver in #44824 uses the same technique as the one in this PR?

The change in this PR is too big in scope.

Yes, I agree that it's too big for merging at once. My intent was to figure out if there's a solution that works for the whole codebase, without any errors, and then implement it incrementally.

It would be a waste to implement some technique only to discover there is a roadblock that prevents us from using it universally.

I think we could change the field module to point to the untranspiled source code (already in ESM), and leave main with the CJS transpilation for published packages.

Beware that there are two kinds of "transpilation" that are orthogonal to each other:

  • what code syntax does the code use? Old compatible ES5 or latest ES2020 that needs to be transpiled for older browsers?
  • what module syntax does the code use? It's possible to create a CJS module with fancy syntax (ES2020, JSX, even custom Babel extensions) or an ESM module with a conservative ES3 syntax.

The CJS and ESM bits will be completely removed by the bundler (webpack), but the remaining code will stay and will be shipped to the destination (a browser or a Node runtime).

@jsnajdr
Copy link
Member Author

jsnajdr commented Aug 17, 2020

I'm not sold on the idea of introducing calypso:main

Few weeks ago, while on leave, I found this article that suggests using the modern exports field for the same purpose: https://twitter.com/_developit/status/1289271769287671809

Looks like something definitely worth looking at.

@github-actions
Copy link

github-actions bot commented May 5, 2021

This PR has been marked as stale due to lack of activity within the last 30 days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Build [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. [Status] Stale
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants