[rush] Support for only installing Rush components needed for the current repo.

[iclanton]

As we've been building out the Rush build cache feature and adding support for cloud-based cache providers like Azure Storage and AWS S3, it's become evident that the SDK packages for Azure and AWS have a large install footprint.

It would be useful to carve these pieces of Rush into separate packages that are only installed (potentially during a repo's rush install step?) when a repo needs them. We already do something similar with Rush's version selector feature.

I think the best way to structure this is to have a central listing of version-lockstepped "feature packages" in rush-lib that code can request, and will be installed in the user's ~/.rush folder. This can work one of two ways:

1. rush install may not depend on any of these packages, and it installs them for the repo's configuration therefore making them available to the rest of Rush
2. When a feature package is requested, a promise is returned that may go out and install the package.

Each approach has upsides and downsides, and will require design work to not introduce circular dependencies, but I wanted to get the ball rolling on figuring this out. We'll probably want to use this when adding support for S3 a cache provider as the AWS SDK is 57MB.

[wbern]

I had similar issues when I wrote a small script to run to determine what to rush install/build, where I wanted to use one npm dependency. Installing that dependency itself would have lead to a 15 second delay just to be able to run the script, so I opted for writing the logic myself.

In cases like what you describe however, it seems inevitable to have to download some things only when needed, kind of like lazy-loading for the web.

I wonder, would you want to have separate rush npm packages that contain the needed extras? Like rush-aws, for example.

It's hard for me to think of any bigger designs at the moment..

[octogonz]

Here's some possible design alternatives:

1. "plugin" model: Rush is decomposed into NPM packages. You can install the specific packages that you want. Each time you install a plugin, the Rush CLI gets new features that you can now access.

   Advantages:

   • The Rush API can be divided into packages as well
   • We don't need to implement a mechanism for auto-installing packages.

2. lazy install model: The Rush CLI always has every feature available. When you use a new feature for the first time, Rush will automatically install the required component (which makes the command take much longer the first time you run it). For airgapped environments, we could provide a way to preinstall all dependencies.

   Advantages:

   • The end user experience is very simple. Nobody needs to configure "plugins," and plugins don't influence config file design.
   • The downloadable chunks do not need to be meaningful package boundaries. They can be "internal" packages with no public contract. This might make the implementation easier.
   • It is easier to guarantee deterministic installs with this design.

3. Webpack everything. Yarn does this -- the entire tool is one 5MB .js file. It installs insanely fast.

[raihle]

Just want to note that v3 of the AWS SDK is modular and much smaller - but still 15 MB for just the S3 client and dependencies.

[boardend]

Regarding the design alternatives @octogonz suggested, a plug-in architecture based on regular packages would just feel natural to Rush.

Please note that a plug-in architecture could also be beneficial when a user wants to provide a custom implementation within the monorepo as (autoinstaller) package (see #2393 (comment)) or to manage multiple monorepos, as discussed at #2500.

And I don't see why the end user experience from the Rush CLI would suffer from such an architecture. The simple use cases (that would just use the official components) shouldn’t need any more configuration/complexity and a lazy install model would still make sense to keep the Rush installation as quick as possible.

[octogonz]

This could also be a partial solution for #2492.

@HipsterZipster had given feedback recently that:

Please eliminate the keytar dependency somehow. For those of us working in locked-down corp environments that can't easily pull prebuilt native libraries from the internet, this is a major pain. We specifically avoid libraries that require native dependencies unless absolutely necessary. Rush's pure JS approach was one of our big draws!

Ideally we should eliminate the keytar dependency entirely. But if we can't do that, a selective install could eliminate keytar at least for people who are not using the build cache feature.

CC @mikeharder (who opened the upstream issue Azure/azure-sdk-for-js#13950 )

[elliot-nelson]

As a user, the "plugin" approach is really appealing to me -- for example, if the docs explained that naming a project "plugins" and adding a package.json with various packages in it would automatically include any packages in it as part of the rush runtime.

(So if I was using the S3 build cache, or even a custom build cache, just add that package to the plugins project and it's good to use.)

As a contributor, the question with any plugin model is how to make it work without ending up with a giant list of abstract "extension points" and factories ala Jenkins...

[m1heng]

As rush has widely adopted in Bytedance across multiple groups, we are also looking for a way to share common logic (like useful commands) and custom cache implementation based on our own object storage.
I think "plugin" feature would definitely help us and rush itself to be more popular in large crop.

[m1heng]

May just make cache functionality as "plugin" mode first would make this issue currently doable.

[dmichon-msft]

I favor the plugin model; we're doing decently well with them in Heft right now. The tapable hooks model retains solid performance and enumerating the plugins to install in the config file is very user-friendly.

I envision having a list of plugins in rush.json, which get installed as part of the normal rush install using your configured package manager. Ideally the rush bootstrapper and installation runtime should aim to have 0 dependencies (though it may need to install your chosen package manager).

If the CLI command does not match "rush install", "rush update", (plus arguments), then rush will load all your configured plugins (some of which will affect the CLI parser), then parse the command and execute it. All plugins should be designed to load only a stub initially and defer load their logic in the event that they are actually used, so as to minimize their impact on boot performance.

One side benefit here is that the core rush runtime would then tend to have a very stable version, and the rush plugins would roll into your lockfile like everything else in the repo.