Infer project references from common monorepo patterns / tools

[RyanCavanaugh]

Genesis: see section "Repetitive configuration" in #3469 (comment)

Search Terms

monorepo infer project references automatically yarn lerna workspace package.json

Suggestion

For common monorepo managers, we should natively understand cross-project references declared in package.json as if they were declared in tsconfig.json

Open questions:

• Which formats (lerna, yarn, pnpm, etc) would be supported? Can all of them be consistently detected, or would you need to opt in to a specific "monorepo format" to enable a specific resolution algorithm?
• How do we find the tsconfig.json file? This data is actually not present in the current (non-tsconfig) dependency graph. We could assume it to be in the package root; what if it's elsewhere?
• Would you need to opt in? Would there be a way to opt out? What should that look like?

Use Cases

• Monorepos of all (supportable) flavors

Examples

https://github.com/RyanCavanaugh/learn-a

This repo has a fair bit of duplication where projects need to write down their dependencies in package.json and as references (with different syntax) in tsconfig.json.

Checklist

My suggestion meets these guidelines:

• This wouldn't be a breaking change in existing TypeScript / JavaScript code
• This wouldn't change the runtime behavior of existing JavaScript code
• This could be implemented without emitting different JS based on the types of the expressions
• This isn't a runtime feature (e.g. new expression-level syntax)

[weswigham]

Which formats (lerna, yarn, pnpm, etc) would be supported? Can all of them be consistently detected, or would you need to opt in to a specific "monorepo format" to enable a specific resolution algorithm?

All the tools you've listed use normal commonjs module resolution (and consequently use the normal package.json dependencies, optionalDependencies, and to a lesser degree devDependencies fields) - none of them change runtime behavior at at all in that regard. Where they differ is where they symlink things into/from (some install all packages in one dir, then link them into each other; some install everything in a top-level node_modules, since that makes everything available), which is what causes issues for us, usually - our symlink handling needs to be essentially perfect to handle all of these configurations correctly. All that'll solve that is hammering away at it with all the configurations we can find and fixing the bugs we find, IMO.

One of the things I was running into in trying to setup an ui-fabric user suite test (where I symlinked things together in our harness rather than use rush) was that depending on where I symlinked stuff, TS would locate modules in ways I wasn't always prepared for. For example: If I symlink from packages/foo into packages/bar/node_modules, imports from packages/foo/index.d.ts is still going to resolve up from packages/foo, not packages/bar/node_modules/foo (which complicates where you need to place things a bit). These tools all usually handle all these cases right (they have to or the runtime wouldn't work); we just need to continue to be faithful to the commonjs resolver's behavior.

How do we find the tsconfig.json file? This data is actually not present in the current (non-tsconfig) dependency graph. We could assume it to be in the package root; what if it's elsewhere?

I would assume package root, and if we don't find one there, we could read a tsconfig field in the package.json that points at it (cue people asking us to read the configuration straight from the package.json).

Would you need to opt in? Would there be a way to opt out? What should that look like?

The usecase is a (near) zero-config start for common monorepo setups, so opt-out, IMO. Could add something like a --no-pkg flag to tsc -b that stops it from walking package.json files.

[bajtos]

How do we find the tsconfig.json file? This data is actually not present in the current (non-tsconfig) dependency graph. We could assume it to be in the package root; what if it's elsewhere?

In https://github.com/strongloop/loopback-next, we are using monorepo to develop a bunch of modules. Right now, we have one tsconfig.json file per each package. (Because of the way how we are working around missing support for project references, we call this file tsconfig.build.json.)

IMO, it's important to allow each package to have its own tsconfig configuration, because this configuration often involves a list of files to include/exclude from compilation.

Assuming the tsconfig.json file is located in the package root makes perfect sense to me 👍

Initially, I was reading the proposal as to assume a single tsconfig.json file located in monorepo root, that would not work (at least for us).

Would you need to opt in? Would there be a way to opt out? What should that look like?

If we can find an elegant way how to support most of commonly-used monorepo layout (tools) that are setting up cross-package dependency tree using the information from standard npm/package.jsondependencies, optionalDependencies and devDependencies fields only, then I think TypeScript's build mode should automatically infer project references from package.json too.

I think the tricky question we need to answer first: how to distinguish between dependencies that are considered as monorepo-local and should be configured as TypeScript references; and external dependencies that should be consumed as read-only? The package.json file does not provide any hints on that.

The first solution that comes to my mind:

• Find out where is the monorepo rooted, either by looking for lerna.json, yarn config file, etc. or by asking the user to explicitly provide that directory via configuration. I think the explicit config option is a better solution as it does not couple TypeScript with different monorepo solutions and provides a natural place for opting into auto-discovery of project references.
• When parsing package.json dependencies, check out which dependencies are resolved as a symlink pointing to a different place inside monorepo - these dependencies should be added as project references. Dependencies symlinked to a different place (outside of monorepo, typically /user/local/lib/node_modules when using npm link manually) (*) or installed as a copy in node_modules should be consumed as read-only.

I find this solution a bit too complex and involved, but don't have any better alternative right now :(

(*) Handling of symlinks outside of monorepo is actually another interesting case to discuss. If we treat all symlinked dependencies as project references, then people using multiple single-repo projects with manual npm link could get benefits from the new build mode too.

For example, loopback depends on strong-remoting, where both modules are maintained by the same team. Sometimes I need to make changes both in loopback and strong-remoting, where the change in loopback depends on the changes made in strong-remoting. To do that, I run npm link path-to-my-strong-remoting-clone in my loopback directory. If we were using TypeScript and the build mode was treating all symlinks as project references, then I could make cross-project renames and rebuild both loopback & strong-remoting in a single build step.

[weswigham]

I think the tricky question we need to answer first: how to distinguish between dependencies that are considered as monorepo-local and should be configured as TypeScript references; and external dependencies that should be consumed as read-only?

I was thinking we could update the -b flag to accept a list of globs (whereas today it takes a single folder or list thereof), so it works like the packages field in lerna or workspace fields in npm/yarn, so you can easily pass tsc -b packages/* apps/* to say "all the packages in these two folders are part of my build". I've found that to be the concise way to express how I've seen these repos laid out when I've been comparing rush, learna, and workspaces - of those, rush is the only one that I've seen be regularly much more explicit than that in its config. IMO, glob support in the input paths are probably worthwhile even if you'd not use package.json reading.

[mhegazy]

how would that work for tsserver?

[weswigham]

@mhegazy AFAIK Nothing in tsserver yet cares about project references (or the dependency graph thereof), just the presence of declaration maps. 😉 The only one that requires it is probably a cross-project compile on save - which is a matter of integrating the entire -b flag into the server - arguments included in some way. Which, for that, we could choose to interpret a top-level tsconfig similar to

{
  "references": [{ path: "packages/*" }]
}

to setup the context. (Which, hopefully, would also allow tsc -b in that directory to work without any arguments). This mimics a lerna.json, a rush,json, or the top-level package.json used for npm/yarn workspaces.

[weswigham]

I mean, jamming it into a tsconfig is ultimately wrong (a .tsbuildrc.json with dedicated build-context wide settings would be more appropriate), since tsc -b's arguments don't correspond to tsc's normal arguments at all, so re-purposing tsc's config file is also wrong (since 99% of it would be meaningless and the remaining 1% would be repurposed overlap), but I've lost that fight already.

[weswigham]

Or alternatively, we can just actually read a lerna.json, a rush.json (maybe), or a top-level package.json with a workspaces field; because why duplicate even that configuration when you don't need to.

[Cryrivers]

I think cross repo reference + yarn workspace worked before TypeScript 2.9

Say you want to import something from your monorepo folder packages/@common/library，the code fix feature in VSCode could correctly suggest the path @common/library in TypeScript 2.8.

However, since 2.9, I believe it goes through symlinks or something, VSCode now suggests something like ../../../../../../node_modules/@common/library, resolving all the way back to the node_modules folder in monorepo root (with yarn workspace enabled).

[timfish]

Also bear in mind that the various tools can treat the workspace globs differently.

I tried pnpm recursive install which goes looking through every subdirectory whereas other monorepo tools I've tried only look at the first level of directories. That meant it also went through a jspm_packages directory we have and tried to install and symlink all those too!

The downside to trying to support specific monorepo types is that these are just the ones we're using this week. Next week we'll probably be using something completely different 😆

[friflo]

I am using a monorepo setup based on lerna. The problem I have in this scenario is, that the root folder contain all dependencies of all packages.
This means that every *.ts file is able to import any module, which is in root/node_modules.
Before I made this experience I expected that only the package.json dependencies are considered to be resolved. This is what a developer would expect.
I think it is sufficient to apply this type of module resolution at compile time via a config setting in tsconfig.json (e.g. "dependencies": "./).
The module resolution at runtime (via node) should be unchanged.

[dicarlo2]

@Cryrivers actually, that bug was introduced in 2.9.2. There's an issue open for it, but I can't seem to find it now. 2.9.1 works as expected.

[akosyakov]

Just to share: for Theia I've added a script that converts yarn workspaces to ts project references: https://github.com/theia-ide/theia/blob/b7471470214533912174fa1c0b07301346026939/scripts/configure-references#L19

It can be executed as prepare script to make sure that they stay in sync: https://github.com/theia-ide/theia/blob/b7471470214533912174fa1c0b07301346026939/package.json#L50

[Ciantic]

I think this setup is a bit backwards, why do we need to define project references in tsconfig.json when they are already in package.json? Is the purpose of this issue to allow "jump to definition/declaration" to work without declaration maps in monorepo/lerna setups?

Imagine following Lerna monorepo setup:

• packages/components
• packages/adminapp (depends on components)

Each has it's own tsconfig.json, e.g. for adminapp/tsconfig.json:

{
    "extends": "../tsconfig.settings.json",
    "compilerOptions": {
        "outDir": "lib",
        "rootDir": "src"
    }
    "references": [{ "path": "../components" }] // I propose to replace this with more generic packages below
}

Adminapp has the project reference to the components package in the package.json:

{
    // ...
    "dependencies": {
        "@yourcompany/components": "^0.1.1",
    }
}

I propose that instead of having project references, we would have package directories, e.g. In short, instead of this in tsconfig.json:

{
    "references": [{ "path": "../components" }]
}

One could define just the package root directories in tsconfig.json instead:

{
    "packages": ["../"]
}

TS Could look in to that packages directory and search for */package.json files. It would see that there is already package defined in ../components/package.json that provides the @yourcompany/components.

Running a tsc -b -w packages always in the background for declaration maps is a bit annoying, given that setups using webpack/parcel/fuse-box does not even require running tsc in the first place cause they do the building in custom scripts.

It seems a bit odd that monorepo setup can't jump to declaration without a declaration maps. The source code of each project is lying in the packages/PROJECT_NAME/src/ as defined by the tsconfig.json setting rootDir (src/), if there were a way for TS to just look the source code first it shouldn't need to have a declaration maps?

[RyanCavanaugh]

Due to the most common monorepo tool effectively blocking our use of it [1], I don't think we'll be able to do any work here without inviting potential legal problems.

FYI @satyanadella

[1] lerna/lerna#1616

Edit: We're back; see below.

[yortus]

@RyanCavanaugh I think it's premature to close this issue. There must be ways of reducing the duplication of dependency info that aren't tightly coupled to any particular monorepo tool.

We only really need two things, neither of which depend on external monorepo tools:

1. the project dependency graph. This can be computed from the package.json files (that's how the external tools already compute it)
2. a set of glob patterns to identify candidate projects. This can be specified once at the monorepo root, for example in tsconfig.json.

[chriseppstein]

@RyanCavanaugh You can lock to an older version of lerna for now. The license change isn't retroactive.

[RyanCavanaugh]

I'm not interested in getting sued (or getting my employer sued) because I forgot to put the right characters in a package.json I wrote as part of my job. Could have happened once already.

Regardless, any feature work we do relating to lerna would need to be tested on its latest version. People will log bugs and point us at their repos that point to a latest version of it, possibly recursively in a way that we won't notice. Just running a customer repro for a bug involving lerna will potentially put us at legal risk, because Microsoft will be "using" software it's forbidden from using.

We can re-open this if a maintained ecosystem that is legal for us to test with opens up.

[yortus]

Well the way I see it, if --build mode is useful on its own merits without depending on monorepo tools, then its approach to dependency declarations can be improved without reference to monorepo tools.

The argument here sounds like "we can't consider fixing or improving aspects of --build mode, because some users with issues or suggestions might be using lerna."

[RyanCavanaugh]

This issue specifically is basically "make tsbuild work from the same fields lerna does" even though that's not the title. If lerna and some tool X (let's call it free lerna, or flerna) standardize on behavior so that we can test flerna without getting sued, then that's all well and good. If not, then some other part of the TS ecosystem written by someone not under a bill of attainder will fill that gap.

My current take on the monorepo ecosystem is that there isn't something as popular as lerna yet, and we don't usually want to invest too much in places where we wouldn't be supporting the most-popular variant as a first-class citizen. Tool popularity will almost certainly change in the future and, like all other issues here, we'll reconsider it at that time even if this particular currently has the open/close bit set to zero.

There is plenty in the pipe to fix and improve --build already. It's not going anywhere.

_{Also I am trolling a tiny bit because come on}

[yortus]

Okaaay. Well can we still look at making project dependency declarations less repetitive? Does it need a new issue? I've already had colleages get the tsconfig.json references out of sync with the package.json deps and it leads to annoying problems (which I'm happy describe in a suitable issue).

Also:

People will log bugs and point us at their repos that point to a latest version of [lerna] ... will potentially put us at legal risk

this is gonna happen anyway, it's not specific to this issue. Hopefully that bit was trolling 🤞.

[SimenB]

npm has announced that they'll support the same workspace feature that yarn supports: https://blog.npmjs.org/post/186983646370/npm-cli-roadmap-summer-2019. Would be sweet if tsc could understand it as well, seems to be accepted as standard at this point

[Bnaya]

yarn has yarn workspaces info command that outputs the needed info to automatically setup references
For example:

 "@thi.ng/rstream-csp": {
   "location": "packages/rstream-csp",
   "workspaceDependencies": [
     "@thi.ng/csp",
     "@thi.ng/rstream"
   ],
   "mismatchedWorkspaceDependencies": []
 },
 "@thi.ng/rstream-dot": {
   "location": "packages/rstream-dot",
   "workspaceDependencies": [
     "@thi.ng/rstream"
   ],
   "mismatchedWorkspaceDependencies": []
 },
 "@thi.ng/rstream-gestures": {
   "location": "packages/rstream-gestures",
   "workspaceDependencies": [
     "@thi.ng/api",
     "@thi.ng/rstream",
     "@thi.ng/transducers"
   ],
   "mismatchedWorkspaceDependencies": []
 },

[Bnaya]

I've created a cli tool based on information from yarn workspaces info (can be made to work with lerna),
That injects the needed refs, set composite: true and also includes other automations on tsconfigs on monorepo
https://www.npmjs.com/package/typescript-monorepo-toolkit

[btakita]

I solved this using a custom shell script. I have similar monorepo project to @thi.ng/* called @ctx-core/*. I tend to use a git submodule to include the packages in active development. @Bnaya, I'm excited about @thi.ng. I'd love to find a way to mix in other large scale monorepo libraries together so we can utilize each others work. I'm sure patterns will emerge. A vision that I see is every developer & organization could cultivate their own monorepo, with a la carte libraries that are under their own control by utilizing forking.

#!/bin/sh
ROOT_DIR=$(dirname $(dirname "$0"))
tsc -b \
  $(ls $ROOT_DIR/packages/ctx-core/packages/*/tsconfig.json | xargs dirname) \
  $(ls $ROOT_DIR/packages/*/tsconfig.json | grep -v ctx-core | xargs dirname) \
  $@

[Bnaya]

@thi.ng is not mine, i'm just making some minor contributions:)
yarn 2 supports nested workspaces. not sure exactly how its gonna work, but for sure its gonna be cool

[Bessonov]

For my personal needs I wrote a tool for pnpm which syncs package.json dependencies to tsconfig.json. Basic idea: look at the workspace and dependencies and if the dependency is a link to one of the workspace module, then put relative path in tsconfig.json. It has an alpha quality, but works great for small/middle sized monorepo: https://github.com/Bessonov/set-project-references

[azu]

📝 Its note from my experience of creating @monorepo-utils/workspaces-to-typescript-project-references.

We can use the almost same sync logic to following package managers.

• npm 7 beta supports workspaces field in package.json
• Yarn v1 supports workspaces field in package.json
• Yarn v2(berry) also supports workspaces field in package.json
  • Yarn v2 will introduce Workspace ranges workspace:, but it is experimental
  • It works as an alias feature. it is a bit difficult to infer the correct reference.

There is a package manager/monorepo tool that uses another logic.

• pnpm use pnpm-workspace.yaml
• Bolt use bolt.workspaces
• etc..

📝 lerna use lerna.json, but it respect package manager logics. e.g. yarn workspaces support

It is difficult to support All monorepo tools because currently does not exists workspaces specification.
(@monorepo-utils/workspaces-to-typescript-project-references has introduced plugin feature for resolving this issue.)

However, workspaces field may be defacto standard after npm v7 would be released.

If TypeScript supports workspaces field, it fills basic needs, I think.

[spion-h4]

What is the opinion on this issue at the moment? Is the view that --incremental makes this unnecessary, or is the view that we still don't have a good enough option?

IMO this is still a major pain point for TS - it all works fine when its a single package, but almost any project nowadays involves some kind of monorepo setup... The methods people seem to be using to solve this

• baseUrl and paths - can subtly fail (bonus points to whoever can guess how)
• autogenerate project references - pre-build steps painful to set up
• not use project mode (just lerna/wsrun plus possibly the incremental option)
• bazel (!)

[pleunv]

I'm looking into converting a few existing repositories into a pnpm workspace monorepo as well (+ possibly add rush to the mix if I get pnpm to work) and I've been hitting my head against this for the past few days.

There's various articles (and issues on SO) to be found covering monorepo setups with TypeScript and lerna or yarn. However, most of these are outdated, contain conflicting information or do things in different ways, in addition to having various DX shortcomings (i.e. no proper @scope auto-import, no definition lookups, manual building of libs, and so on) that honestly make the whole monorepo setup a no-go for me.

There doesn't really appear to be any official documentation anywhere merging all of these concepts together, never mind adding something like babel-preset-typescript into the mix. There's definitely parts of this setup that are package manager or bundler specific, but it has also been getting increasingly difficult to keep up with and figure out how various aspects of the TypeScript config interact with one another in a (nested) project structure like this. I'm thinking about things like project references, paths, incremental builds & composite, noEmit, declarations & maps, and so on. There's also VSCode's new multi-level root workspaces that I've seen mentioned here and there but that honestly doesn't seem to make much of a difference for me at all.

My conclusion after these past few days is that the level of in-depth TS knowledge you seem to need to get a setup like this at least semi-functional just feels too high (read as: I couldn't figure it out and I'm rather frustrated by it). This is not a critique to the TS team, I'm just wondering if what I've been trying to set up is simply not feasible at the moment. I'm also not really sure where this belongs. Workspaces are currently implemented by package managers, but TypeScript seems to be moving more and more into a "holistic" role, adding various capabilities to link different (nested) projects together, so there's definitely quite a bit of overlap.

Since it's seemingly going to take a while before package managers converge on a workspace approach, making native workspace support inside TS rather difficult, could it perhaps be an option to gather & document various approaches in use today, with their benefits and (DX) shortcomings? Or document a section with best practices somewhere?

[jaredpalmer]

Causing auto import to fail: vercel/turborepo#331

[pokey]

Probably not as good as a single source of truth, but fwiw meta-updater can be used to keep them in sync. See the config that pnpm uses in their own monorepo

[trikadin]

Is it possible to implement something like this?

tsconfig.json

{
  "references": [
    {
      "path": "workspace:@namespace/package-name[/tsconfig.json]"
      // or "workspace": "@namespace/package-name"
    }
  ]
}

package.json of "@namespace/package-name"

{
  "exports": {
    "tsconfig.json": "path/to/tsconfig.json"
  }
}

Advantages:

1. Explicit Control: Developers can explicitly manage which references they want to include in the build.
2. Minimal Changes: No additional fields in package.json and almost no new logic required.
3. Ease of Implementation: Likely straightforward to implement.

[jtwebman]

I was thinking lets add a new compilerOptions called workspace. It defaults to none but supports npm, yarn, pnpm. All it does is add to the references based on the rules of that workspace tool. I am just now starting to read though the Typescript compiler and code so not sure what that would entail but it seems easy to me. The advantage here is it can look at the projects deps and dev deps and then see if those projects are workspaces in the current project. If so add them. If not don't. That way you don't have globs which will not work if you have circler references. Will fork and give it a try but what is the process to get this as a feature request and into the code? Does Microsoft except pull requests?