Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: Add npm workspaces #103

Closed
wants to merge 11 commits into from
Closed

RFC: Add npm workspaces #103

Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
fe71372
RFC: Add npm workspaces
ruyadorno Feb 14, 2020
e8221f4
Update accepted/0000-workspaces.md
ruyadorno Feb 17, 2020
a350c72
Update accepted/0000-workspaces.md
ruyadorno Feb 17, 2020
2596463
Update accepted/0000-workspaces.md
ruyadorno Feb 19, 2020
c279a49
Question: opt-out config?
ruyadorno Feb 20, 2020
de8d71c
Incorporated feedback from the deep-dive call
ruyadorno Mar 11, 2020
25b1683
Add feedback about run-scripts
ruyadorno Mar 17, 2020
102660c
Moved run commands to its own RFC
ruyadorno Mar 19, 2020
3335cf7
Added ref to negative globs config
ruyadorno Mar 19, 2020
8a5080c
Clean up example title
ruyadorno Mar 19, 2020
26e8ac6
Removed unresolved questions
ruyadorno Mar 26, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
291 changes: 291 additions & 0 deletions accepted/0000-workspaces.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,291 @@
# npm workspaces

## Summary

Add a set of features to the **npm cli** that provide support to managing multiple packages from within a singular top-level, root package.

## Motivation

This feature has been requested by the community for a long time. The primary motivation behind this RFC is to fully realize a set of features/functionality to support managing multiple packages that may or may not be used together.

The name “workspaces” is already established in the community with both Yarn and Pnpm implementing similar features under that same name so we chose to reuse it for the sake of simplicity to the larger community involved.

## Detailed Explanation

After sourcing feedback from the community, there are 2 major implementations/changes required in the **npm cli** in order to provide the feature set that would enable a better management of nested packages.
- Make the **npm cli** **workspace**-aware.
- Install: In a **npm workspaces** setup users expect to be able to install all nested packages and perform the associated lifecycle scripts from the **Top-level workspace**, it should also be aware of **workspaces** that have a **dependency** on one another and **symlink** them appropriately.

The set of features identified in this document are the ones that are essential to an initial [MVP](https://en.wikipedia.org/wiki/Minimum_viable_product) of the **npm workspaces** support. The community should expect further development of this feature based on the feedback we collected and documented at the end of this RFC.

## Rationale and Alternatives

First and foremost there’s the alternative of leaving the problem set for userland to solve, there’s already the very popular project [Lerna](https://github.com/lerna/lerna) that provides some of these features.

Also available is the alternative of supporting only the install (or bootstrap as Lerna names it) aspect of this proposal, following a less feature-rich approach but one that would still enable the basic goal of improving the user experience of managing multiple child packages but from all the feedback collected during the research phase of this RFC, this alternative is much less desirable to the community of maintainers involved.

## Implementation

### 1. Workspaces configuration: Making the npm cli workspace-aware

We're following the lead of Yarn in supporting the `workspaces` `package.json` property which defines a list of paths, each of these paths may point to the location of a **workspace** in the file system but it also support **globs**.

`package.json` example:

```
{
"name": "workspace-example",
"version": "1.0.0",
"workspaces": {
ruyadorno marked this conversation as resolved.
Show resolved Hide resolved
"packages": [
"packages/*"
]
}
}
```

`package.json` shorthand example:

```
{
"name": "workspace-example",
"version": "1.0.0",
"workspaces": [
"packages/*"
]
}
```

The **npm cli** will read from the paths and **globs** defined in this **workspaces configuration** and look for valid `package.json` files in order to create a list of packages that will be treated as **workspaces**.

**Note:** The `packages` property should support familiar patterns from [npm-packlist](https://www.npmjs.com/package/npm-packlist) `files` definition such as negative globs.


### 2. Installing dependencies across child packages

Change `npm install` ([arborist](https://www.npmjs.com/package/@npmcli/arborist)) behavior to make it properly install **dependencies** for every **workspace** defined in the **workspaces configuration** described above.

**Arborist** should also be aware of all **workspaces** in order to correctly link to another internal **workspace** should it match the required semver version of an expected **dependency** anywhere in the installing tree. e.g:

```
// Given this package.json structure:
├── package.json { "workspaces": ["dep-a", "dep-b"] }
├── dep-a
│   └── package.json { "dependencies": { "dep-b": "^1.0.0" } }
└── dep-b
   └── package.json { "version": "1.3.1" }

$ npm install

// Results in this symlinking structure:
├── node_modules
│   ├── dep-a -> ./dep-a
│   └── dep-b -> ./dep-b
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC, this is proposing to link all sub-packages to root-level node_modules folder. I find such arrangement as rather problematic, because it allows sub-packages to require other sub-packages that are not listed in the dependencies. Such code will work at runtime when executed inside the monorepo (e.g. when running tests of the package), but will fail once the problematic package is installed from the registry (i.e. when consuming the package in 3rd-party projects).

As an example, consider the example presented in the RFC document, with an additional dep-c package with no dependencies and a file dep-c/index.js that's calling require('dep-a').

Now when you run node -e "require('dep-c')" from monorepo root, all is good. When you run the same command from a project depending on dep-c, it will fail with Error: Cannot find module 'dep-a'.

In lerna, workspace sub-packages are never linked to root-level node_modules, the symlinks are always created between the packages only.

├── node_modules
├── dep-a
│   └── node_modules
│       └── dep-b -> ../../dep-b
├── dep-b
└── dep-c

As a result, node -e "require('dep-c')" executed from monorepo root fails.

I understand that hoisting all sub-packages in a shared top-level directory may have benefits. In that case I am proposing to use a different directory name than node_modules, e.g. .workspace_modules.

├── .workspace_modules
│   ├── dep-a -> ./dep-a
│   └── dep-b -> ./dep-b
├── dep-a
│   └── node_modules
│       └── dep-b -> ../../../.workspace_modules/dep-b
└── dep-b

├── dep-a
└── dep-b
```

For the initial **workspaces** implementation, we're going to stick with **arborist**'s default algorithm that privileges **hoisting packages** but will place packages at nested `node_modules` whenever necessary.


## Examples

### Expanding on symlinking structure and `package-lock` file shape.

Given a **npm workspaces** setup with the following contents:

```
$ cat ./package.json
{
"name": "foo",
"version": "1.0.0",
"workspaces": [
"./core/*",
"./packages/*"
],
dependencies: {
"lodash": "^4.x.x",
"libnpmutil": "^1.0.0"
}
}

$ cat ./core/libnpmutil/package.json
{
"name": "libnpmutil",
"version": "1.0.0",
"dependencies": {
"lodash": "^4.x.x"
}
}

$ cat ./packages/workspace-a/package.json
{
"name": "workspace-a",
"version": "1.7.3",
"peerDependencies": {
"react": "^16.x.x"
},
"dependencies": {
"workspace-b": "^2.0.0"
}
}

$ cat ./packages/worskpace-b/package.json
{
"name": "workspace-b",
"version": "2.1.1",
"peerDependencies": {
"react": "^16.x.x"
}
}

$ cat ./packages/workspace-c/package.json
{
"name": "workspace-c",
"version": "1.0.0",
"peerDependencies": {
"react": "^16.x.x"
},
"dependencies": {
"workspace-b": "^1.0.0"
}
}
```

Will result in the following symlinking structure:

```
$ tree
.
├── package-lock.json
├── node_modules
│   ├── lodash
│   ├── libnpmutil -> ./core/libnpmutil
│   ├── workspace-a -> ./packages/workspace-a
│   ├── workspace-b -> ./packages/workspace-b
│   ├── workspace-c -> ./packages/workspace-c
│   └── react
├── core
│ └── libnpmutil
└── packages
├── workspace-a
├── workspace-b
└─ worspace-c
   └── node_modules
   └── workspace-b@1.0.0
```

And the following `package-lock.json` files:

NOTE: The following lockfile is for illustration purpose only and its final shape might differ.

```
$ cat ./package-lock.json
{
"name": "foo",
"version": "1.0.0",
"lockfileVersion": 2,
"requires": true,
"packages": {
"": {
ruyadorno marked this conversation as resolved.
Show resolved Hide resolved
"name": "foo",
"version": "1.0.0",
"dependencies": {
"lodash": "^4.17.15",
"libnpmutil": "^1.0.0"
}
},
"core/libnpmutil": {
"name": "libnpmutil",
"version": "1.0.0"
},
"node_modules/lodash": {
"name": "lodash",
"version": "4.17.15",
"resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.15.tgz",
"integrity": "sha512-8xOcRHvCjnocdS5cpwXQXVzmmh5e5+saE2QGoeQmbKmRS6J3VQppPOIt0MnmE+4xlZoumy0GPG0D0MVIQbNA1A=="
},
"node_modules/libnpmutil": {
"resolved": "core/libnpmutil",
"link": true
},
"node_modules/workspace-a": {
"resolved": "packages/workspace-a",
"link": true
},
"node_modules/workspace-b": {
"resolved": "packages/workspace-b",
"link": true
},
"node_modules/workspace-c": {
"resolved": "packages/workspace-c",
"link": true
},
"packages/workspace-a": {
"name": "workspace-a",
"version": "1.7.3"
},
"packages/workspace-b": {
"name": "workspace-b",
"version": "1.0.0"
},
"packages/workspace-c": {
"name": "workspace-c",
"version": "1.0.0"
}
},
"dependencies": {
"lodash": {
"version": "4.17.15",
"resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.15.tgz",
"integrity": "sha512-8xOcRHvCjnocdS5cpwXQXVzmmh5e5+saE2QGoeQmbKmRS6J3VQppPOIt0MnmE+4xlZoumy0GPG0D0MVIQbNA1A=="
},
"libnpmutil": {
"version": "file:core/libnpmutil"
},
"workspace-a": {
"version": "file:packages/workspace-a"
},
"workspace-b": {
"version": "file:packages/workspace-b"
},
"workspace-c": {
"version": "file:packages/workspace-c",
"dependencies": {
"workspace-b": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/workspace-b/-/workspace-b-1.0.0.tgz",
"integrity": "sha512-8xOcRHvCjnocdS5cpwXQXVzmmh5e5+saE2QGoeQmbKmRS6J3VQppPOIt0MnmE+4xlZoumy0GPG0D0MVIQbNA1A=="
},
}
}
}
}
```


## Dictionary

During the discussions around this RFC it was brought up to our attention that a lot of the vocabulary surrounding what the larger JavaScript community understands as "workspaces" can be confusing, for the sake of keeping the discussion as productive as possible we're taking the extra step of documenting what each of the terms used here means:

- **npm cli**: The [npm cli](https://github.com/npm/cli/) :wink:
- **npm workspaces**: The feature name, meaning the ability to the **npm cli** to support a better workflow for working with multiple packages.
- **workspaces**: A set of **workspace**.
- **workspace**: A nested package within the **Top-level workspace** file system that is explicitly defined as such via **workspaces configuration**.
- **Top-level workspace**: The root level package that contains a **workspaces configuration** defining **workspaces**.
- **workspaces configuration**: The blob of json configuration defined within `package.json` that declares where to find **workspaces** for this **Top-level workspace** package.
- **dependencies**: A set of **dependency**.
- **dependency**: A package that is depended upon by another given package.
- **dependent**: A package which depends on another given package.
- **symlink**: A [symbolic link](https://en.wikipedia.org/wiki/Symbolic_link) between files.
- **[globs](https://en.wikipedia.org/wiki/Glob_(programming))**: String patterns that specifies sets of filenames with special characters.
- **[Arborist](https://github.com/npm/arborist)**: The npm@7 install library
- **hoisting packages**: Bringing packages up a level in the context of an installation tree.
- **[scripts](https://docs.npmjs.com/misc/scripts)**: Arbitrary and lifecycle scripts defined in a `package.json`


## Prior Art

- [Lerna](https://github.com/lerna/lerna)
- [Yarn workspaces](https://classic.yarnpkg.com/en/docs/workspaces/)
- [Pnpm workspaces](https://pnpm.js.org/en/workspaces)