[RRFC] Query Installs

[jamiebuilds-signal]

Motivation ("The Why")

1. Extending the usefulness of npm query

With the new npm query command it's now possible to select subtrees of your dependencies for many different purposes, auditing, inspecting, upgrading dependencies, funding, etc.

For all of these purposes, you sometimes also want the contents of these dependencies as well, even a full install. You can run a full install today, but that tends to take awhile and if you're auditing or want to do something sensitive

2. Allow users to install just the packages they need for a task

For users with sizable installs, npm install is one of the longest steps in their CI process. And oftentimes its not worthwhile to parallelize running separate npm scripts across CI nodes, simply because npm install takes much longer than the script itself.

If you could run:

npm install --query "eslint, [name^=eslint-plugin-], [name^=@typescript-eslint/], typescript"

And get the subtree of your dependencies queried, it would be much faster than trying to run a full npm install/ci and splitting tasks across ci nodes would be much more impactful.

Other tools could be written around npm install --query to make this more powerful by generating queries for you.

3. "Explain" existing features like --production

Features like --production could be implemented via queries, it could be called an alias.

4. Avoid building one-off features in installs

By giving users a powerful querying syntax in installs, many feature requests related to installs could point to this instead.

Example

npm install --query <query>
npm install --query "*:attr(contributors, :attr([name~=Jordan]))"
npm install --query "eslint, [name^=eslint-plugin-], [name^=@typescript-eslint/], typescript"
npm install --query ".prod" # same as `npm install --production`

How

Current Behaviour

N/A, only options are npm install [--production]

Desired Behaviour

• npm install --query <query>
• Build the minimal tree that includes all of the dependencies matched by <query>
• Preserve npm install's other behaviors otherwise

References

Related to #360, #592

[ljharb]

Queries work on an installed node_modules dir, i thought - it definitely seems like a reasonable addition to npm prune, regardless.

[jamiebuilds-signal]

Arborist has a loadVirtual() function that works off of the lockfile so I presume the logic is close to there already

[ljharb]

True - presumably it could also function off the ideal tree in the absence of a lockfile.

[darcyclarke]

@jamiebuilds-signal love that you see the value of the Dependency Selector Syntax/Queries & 100% +1 your understanding/hope for install to support queries. If you read through the original RFC, you can see that pretty much every npm command is on our radar to augment with --query at some point but it may take a bit to fully virtualize @npmcli/arborist so it can use the query syntax when in a state prior to an initial installation/reification.

Love how you've framed this though & appreciate that you're able to see the value here (it may just take some time to implement).

[rondales]

Motivation ("The Why")

1. Extending the usefulness of npm query

With the new npm query command it's now possible to select subtrees of your dependencies for many different purposes, auditing, inspecting, upgrading dependencies, funding, etc.

For all of these purposes, you sometimes also want the contents of these dependencies as well, even a full install. You can run a full install today, but that tends to take awhile and if you're auditing or want to do something sensitive

2. Allow users to install just the packages they need for a task

For users with sizable installs, npm install is one of the longest steps in their CI process. And oftentimes its not worthwhile to parallelize running separate npm scripts across CI nodes, simply because npm install takes much longer than the script itself.

If you could run:

npm install --query "eslint, [name^=eslint-plugin-], [name^=@typescript-eslint/], typescript"

And get the subtree of your dependencies queried, it would be much faster than trying to run a full npm install/ci and splitting tasks across ci nodes would be much more impactful.

Other tools could be written around npm install --query to make this more powerful by generating queries for you.

3. "Explain" existing features like --production

Features like --production could be implemented via queries, it could be called an alias.

4. Avoid building one-off features in installs

By giving users a powerful querying syntax in installs, many feature requests related to installs could point to this instead.

Example

npm install --query <query>
npm install --query "*:attr(contributors, :attr([name~=Jordan]))"
npm install --query "eslint, [name^=eslint-plugin-], [name^=@typescript-eslint/], typescript"
npm install --query ".prod" # same as `npm install --production`

How

Current Behaviour

N/A, only options are npm install [--production]

Desired Behaviour

• npm install --query <query>
• Build the minimal tree that includes all of the dependencies matched by <query>
• Preserve npm install's other behaviors otherwise

References

Related to #360, #592