[Feature] Provide a package-level cache management library

[arcanis]

Describe the user story

Multiple projects use per-package caching mechanisms, and they often rewrite very-similar-but-ever-so-slightly-different logics. Some write the files directly within each packages (Angular), inside a dot-folder at the root of the node_modules (Babel / Webpack), others are putting it inside a public or build directory (Gatsby, Next), ...

I'd like to provide an all-in-one caching API (shipped under the name @yarnpkg/cache or something similar) that would cover those use cases and would benefit from extra optimizations unlocked by Yarn 2.x. It would also make it easier for third-party projects to stop writing files within the packages (which are now read-only).

Describe the solution you'd like

First, one important point: it goes without saying that this API would have to be compatible with npm and Yarn 1.x. We're not looking to make a Yarn 2.x-specific library, as that would never be adopted anywhere. Any Yarn 2.x-specific feature would have to degrade gracefully on other systems.

The two basic features that this API should provide:

• Given a file path, obtain a Cache instance unique to the given package.
• The Cache instance should expose a public path property pointing to a writable location.

Additionally, this API would take advantage of the way Yarn 2.x keeps files within zip archives by exposing a property indicating whether the source package is likely to see its sources be modified or not. This simple hint would make build processes faster because tools like babel-register will be able to skip the checksum validation for all files that belong to third-party code¹.

So for example, assuming the field is called likelyModified (let's keep the bikeshedding on Discord if possible 😛):

• likelyModified: false would be returned for packages stored in zip archives
• likelyModified: true would be returned for unplugged packages and workspace caches
• likelyModified: true would always be returned if the install strategy is npm or Yarn 1.x

Then tools could shortcut the integrity checks:

const {path, likelyModified} = cache.getPackageCache(sourceFile);
const forceCacheCheck = !!process.env.DEBUG;

if (!likelyModified && !forceCacheCheck)
  return getCachedData(path, sourceFile);

if (checkCacheIntegrity(path, sourceFile))
  return getCachedData(path, sourceFile);

return setCachedData(path, sourceFile);

¹ It isn't possible to make this assumption with npm and Yarn 1.x, because two different packages may have shared the same location on disk after an hoisting reorganization.

Describe the drawbacks of your solution

• This is one more tool we'd have to maintain. It should be fairly well scoped so I don't think the maintenance cost will be very high.

• It would be less risky to spend time on this we can get the greenlight from various open-source projects first.

Describe alternatives you've considered

We could keep things as they are and let projects manage their cache themselves. I think there's an opportunity to make applications and build faster that's worth exploring, though.

[arcanis]

Pinging a few people / projects that might be interested by this approach:

• @gaetanmaisse / Storybook
• @yangshun / Docusaurus
• @nicolo-ribaudo / Babel
• @evilebottnawi / Webpack

Would you be open to discuss a uniform cache mechanism?

[alexander-akait]

There are more popular tools - eslint / stylelint, they also use caches and perhaps would like to speak too