Implement the Yarn Plug'n'Play module resolution algorithm

[evanw]

Node comes with a package manager called npm, which installs packages into a node_modules folder. Node and esbuild both come with built-in rules for resolving import paths to packages within node_modules, so packages installed via npm work automatically without any configuration. However, many people use an alternative package manager called Yarn. While Yarn can install packages using node_modules, it also offers a different package installation strategy called Plug'n'Play, which is often shortened to "PnP" (not to be confused with pnpm, which is an entirely different unrelated package manager).

Plug'n'Play installs packages as .zip files on your file system. The packages are never actually unzipped. Since Node doesn't know anything about Yarn's package installation strategy, this means you can no longer run your code with Node as it won't be able to find your packages. Instead, you need to run your code with Yarn, which applies patches to Node's file system APIs before running your code. These patches attempt to make zip files seem like normal directories. When running under Yarn, using Node's file system API to read ./some.zip/lib/file.js actually automatically extracts lib/file.js from ./some.zip at run-time as if it was a normal file. Other file system APIs behave similarly. However, these patches don't work with esbuild because esbuild is not written in JavaScript; it's a native binary executable that interacts with the file system directly through the operating system.

Previously the workaround for using esbuild with Plug'n'Play was to use the @yarnpkg/esbuild-plugin-pnp plugin with esbuild's JavaScript API. However, this wasn't great because the plugin needed to potentially intercept every single import path and file load to check whether it was a Plug'n'Play package, which has an unusually high performance cost. It also meant that certain subtleties of path resolution rules within a .zip file could differ slightly from the way esbuild normally works since path resolution inside .zip files was implemented by Yarn, not by esbuild (which is due to a limitation of esbuild's plugin API).

With this PR, esbuild now contains an independent implementation of Yarn's Plug'n'Play algorithm (which is used when esbuild finds a .pnp.js, .pnp.cjs, or .pnp.data.json file in the directory tree). Creating additional implementations of this algorithm recently became possible because Yarn's package manifest format was recently documented: https://yarnpkg.com/advanced/pnp-spec/. This should mean that you can now use esbuild to bundle Plug'n'Play projects without any additional configuration (so you shouldn't need @yarnpkg/esbuild-plugin-pnp anymore). Bundling these projects should now happen much faster as Yarn no longer even needs to be run at all. And path resolution rules within Yarn packages should now be consistent with how esbuild handles regular Node packages. For example, fields such as module and browser in package.json files within .zip files should now be respected.

Keep in mind that this is brand new code and there may be some initial issues to work through before esbuild's implementation is solid. Yarn's Plug'n'Play specification is also brand new and may need some follow-up edits to guide new implementations to match Yarn's exact behavior. If you try this out, make sure to test it before committing to using it, and let me know if anything isn't working as expected. Should you need to debug esbuild's path resolution, you may find --log-level=verbose helpful.

[mbrevda]

Release notes say this includes backwards incompatible changes. Does that only apply to users of Yarn pnp?

[evanw]

Yes, basically. Hypothetically also to people with .zip files in their repo.