Monorepo support

[valler]

🔍 Search Terms

skipLibCheck, node_modules, package

✅ Viability Checklist

• This wouldn't be a breaking change in existing TypeScript/JavaScript code
• This wouldn't change the runtime behavior of existing JavaScript code
• This could be implemented without emitting different JS based on the types of the expressions
• This isn't a runtime feature (e.g. library functionality, non-ECMAScript syntax with JavaScript output, new syntax sugar for JS, etc.)
• This isn't a request to add a new utility type: https://github.com/microsoft/TypeScript/wiki/No-New-Utility-Types
• This feature would agree with the rest of our Design Goals: https://github.com/Microsoft/TypeScript/wiki/TypeScript-Design-Goals

⭐ Suggestion

The TypeScript compiler should be aware of package boundaries, especially when it comes to --noEmit type checks.

Basically anything in node_modules should be "none of its business", at least optionally.

The following options give the impression that this is somehow achievable, when in fact it's not doable at all (or maybe only with yet another tool built to work around compiler limitations):

• moduleResolution
• resolvePackageJsonExports
• resolvePackageJsonImports
• skipLibCheck
• isolatedModules
• references in combination with tsc --build --noEmit

Except for browsers, engines these days all support TS in one way or another, and
package managers all support monorepos in one way or another.
During development, there is practically no need (anymore) to compile the source, which would improve DX.
Compilation still is required for browsers, and/or the goal is to reduce shipped code size, or to build a SEA.

However:

Without a compilation step, TSC won't find declaration files and therefore checks source files, treating them as first party code, even if this code is explicitly imported as a third party package (via a specifier that resolves into another package through node_modules, not a relative path that doesn't include node_modules).

Checking of yet uncompiled dependencies in a monorepo will resolve modules from other packages to relative paths / internal dependencies, and therefore a module will be checked as often as it appears as a dependency of a package in the monorepo.

E.g. package a and b depend on c, therefore c will be type checked three times if each package has it's own tsc --noEmit script in its package.json (which it should). That's wasteful, but so is compiling/building dependencies before being able to type check.

Isolated checking of modules and/or packages should be a no-brainer for users. Unfortunately it isn't.

To summarize the feature request:

tsc --noEmit in monorepos should check uncompiled packages only as far as it would with declaration files. The boundary should be the interface imported/used on the call site, without diving deeper into typings of its implementation.

Also note, "monorepo" is mentioned for practical purposes only; what really matters is the package boundary, regardless of monorepo specifics.

The rest are issue references to a subset of those I've read before opening this one, to make sure not to open up an obvious duplicate.

Similar issues, but not focusing on monorepos:

• Feature Request: allow exclusion of node_modules when skipLibCheck is false #30511
• Disable type checking for node_modules entirely #40426
• Add an option to ignore type checking for certain files #52325

Not as closely related, but also highlights on conflicts between tsc and node_modules resolution:

• TypeScript's ecosystem of self-bundled typings does not work with npm #8836

📃 Motivating Example

import "@scope/module"; // should only be checked on the interface when running `tsc --noEmit`
import "#mapped-to-@scope/module"; // same

// To illustrate the boundary:
export const f = (a: string): number => // signature still matters, if reachable according to api exposed by package.json
  a / 2; // implementation inside body: out of scope for packages depending on this export. Yes, this divides a string.

💻 Use Cases

1. What do you want to use this for?

   Type checking packages (or modules) in isolation without compilation. Ideally the fleshed out feature would give you fine grained control over type checker boundaries up to explicit manual definition (think "donut scope" and "globs") in combination with some settings/keywords as presets for well known setups like node packages and npm or pnpm monorepos. The goal is to have as much control over the exits as is given for entries (the latter has --project, files, include, exclude and other options, the former has none).

2. What shortcomings exist with current approaches?

   tsc --emitDeclarationOnly and tsc --noEmit enforce a chronological development hierarchy from dependency to dependent, rather than nonlinear development from interface to implementation across packages. This is not scalable, blocks development and hurts DX, especially in monorepos.

3. What workarounds are you using in the meantime?

   Not using tsc for type checking at all. Only use it for type stripping and/or declaration file generation, which will perform type checks as a side effect unless disabled.

   After some initial discussion: Also tsc --build with references mirroring package.json dependencies and devDependencies further minimises full blown builds, at the expense of manual maintenance of error free mirrors.

[mkantor]

What should happen if instead your function looks like this:

export const f = (a: string): number =>
  a.foo;

And @scope/module's d.ts file contains this:

interface String {
  foo: number
}

?

Currently that typechecks because the compiler follows the @scope/module import and discovers the String augmentation, but I think you're saying that wouldn't happen?

For a more realistic example, what should happen if I install @types/node and then try to use process from my code?

[valler]

It would type check, because that code is part of the interface, meaning f has the ability to use String["foo"].
However the issue is more about uncompiled TS sources, not generated or handwritten declarations.

And regardless of the presence of declaration files, the feature would basically allow the type checker to only check what it has control over. E.g. it will be checked that an imported interface is used correctly, but it won't be checked that the imported interface has a correct implementation. The assumption here is that TS will eventually be stripped down to JS and/or declarations, therefore those type checks will be performed by the other package's type checker according to the other package's config.

[valler]

@mkantor

I've updated the comment in the motivating example from

should not be checked when running tsc --noEmit

to

should only be checked on the interface when running tsc --noEmit

This makes it align with the rest being said and should help avoid confusion.

[RyanCavanaugh]

If it were this straightforward, we would have done it by now 😓. It's very much something we want to happen.

tsc --noEmit in monorepos should check uncompiled packages only as far as it would with declaration files. The boundary should be the interface imported/used on the call site, without diving deeper into typings of its implementation.

This is likely the right kernel of the idea, but quickly you encounter a lot of problems that are much thornier. I will give a rough problem sketch of one of them, but there are others once you start thinking really deeply about it.

Let's consider a super basic example: a depends on b, but b's tsconfig has lib:dom and a's doesn't. In a we write

import { someFunc } from "@myscope/b/someFile.js";
const p = someFunc();
// p: ?

in b we have someFile:

const someArr = [1, 2, 3, 4];
export function someFunc() {
  if (typeof window !== "undefined" && someCondition) {
    window.focus();
    return window.location.href;
  }
  return someArr[3];
}

In the context of a's compilation, window does not exist. This means:

• We have no idea what the return type of someFunc should be, since we don't know what window.location.href is. Absent any special logic, this would be any, which is obviously a disaster. Falling back to number is arguably even worse, since it's extremely conceivable that the first return truly is reachable in at least some context of a's execution.
• We don't even know for sure if that return is reachable since we don't know window.focus() isn't a never-returning function (or an assertion function)

Moreover:

• b's tsconfig could specify options like noUncheckedIndexedAccess differently from a, which means undefined might or might not be in the return type here in a way that differs from what you'd see in an output .d.ts file
• b might import a module that has a global augmentation of some kind that changes its type computations

In short, it's really only provably safe to treat someone else's .ts file as it if were in your compilation when:

• They have identical compiler settings
• They have identical global-affecting files loaded
  • And these files are in the same order (!)

But, to a first approximation, these conditions are never met. And worse, you really don't know if they're met or not, because how do you know whether or not b would have identical global-affecting files loaded? You have to go and resolve all of its imports, check its tsconfig as it if it were a fresh progrm load, etc..

One option, in the face of this problem, is to just sort of give up, throw everything into a giant ball of code, and ignore the fact that this will often generate global conflicts (example: two sub-libraries have different dependency versions on some globally-loaded variable). This is possible today and some teams actually do it, but it offers a much lower threshold of correctness, and absolutely will miss some errors that you would see doing a per-project compilation (e.g. referring to a global variable or module that is not actually resolvable from your config).

Another option is to say "you can't have inference that depends on compiler settings", which is just isolatedDeclarations. It's reasonably tempting to provide some setting where, when you cross the node_modules boundary, you enter isolatedDeclarations mode and refuse to do inference (perhaps even not parsing function bodies, to save AST memory). This is mostly correct, but still falls victim to the fact that two different programs referring to a bare free identifier might in reality be referring to different global types, but this is usually already a problem under declaration emit (you and your upstream project need to agree somewhat about your global environment, after all). This is the most promising approach but still has a lot of downstream issues to iron out, e.g. what happens if you do encounter an unresolved name in this program? There's a lot to think about here.

[valler]

Thanks for the detailed explanation @RyanCavanaugh 😄

If it were this straightforward, we would have done it by now 😓. It's very much something we want to happen.

No doubt, that's why this says "for users" 😉 :

Isolated checking of modules and/or packages should be a no-brainer for users. Unfortunately it isn't.

Yes, something towards isolatedDeclarations is one of the things I had in mind as well (that's why the motivating example has an explicit return type); and yes, agreement on globals and compiler settings is a problem that reaches even further, but also should be less of a problem for the use case of a monorepo, where agreement among the repo's packages should be achievable by design and reduce the problem to the common case of first-to-third-party-conflicts.

You already mentioned the example given not to be exhaustive, but to take it from there, with someFunc being part of a package interface, its return type should be compatible with isolatedDeclarations to some extent. I don't know if I would go as far as disabling inference all together. Probably even the opposite. I'd suggest the compiler should make --noEmit (or a new flag, perhaps --check) do everything that needs to happen to achieve efficient type checking (in monorepos). Or if that's already possible, then that should be documented and streamlined into a single flag. [Edit (to clarify): less error messages along "you forgot a return type", more automatic, yet efficient, code/declaration generation, so users have to write and configure less, without sacrificing type safety].

E.g. the initial pass emits declarations; every subsequent pass performs only checks (where no build is required). From a user's perspective what matters most is to avoid redundant compilation and checks in a user friendly manner: No redundant compilation, no redundant checks, no redundant dependency declarations, no self-built script pipelines to glue together the right combination/order of --build, --noEmit invocations.

A (proposed) tsc --check should behave somewhat like tsc --build and/or tsc --build --watch, with the difference being that one really wants to drop the redundancy of having to declare exports and dependencies in package.json and in tsconfig.json references.

That being said, build order in monorepos is also kind of a solved problem tsc won't have to worry about. To achieve error free builds in npm workspaces, one de facto has to list workspaces in a topologically sorted variant. Other tools have their own explicit solutions as well. That's why references is kind of unattractive in a monorepo. Other than that, it's perhaps the closest to a useful solution to this problem.

With that in mind, there's maybe potential for a monorepo friendly build mode that drops references for package.json dependencies.

[RyanCavanaugh]

E.g. the initial pass emits declarations; every subsequent pass performs only checks (where no build is required).

You're describing tsc -b as it exists today (plus automatic inference of references), as far as I can tell?

[valler]

You're describing tsc -b as it exists today, as far as I can tell?

No, because of references. Having to copy package dependencies into references is tedious and error prone.

Edit:

You're describing tsc -b as it exists today (plus automatic inference of references), as far as I can tell?

Yes I do, but the "plus" part is very important 😄

[valler]

@RyanCavanaugh I've added the following to OP's workarounds, to keep it as informative as possible without having to read the full thread:

After some initial discussion: Also tsc --build with references mirroring package.json dependencies and devDependencies further minimises full blown builds, at the expense of manual maintenance of error free mirrors.

[valler]

How about this:

Update the composite config to an enum false, true, package (monorepo ?) to get rid of references and grab the information from package.json (including the root workspace package.json if needed).

The result would be one build per module, followed by checks only, as long as tsbuildinfo says so. For fresh monorepos this would require only a minimal implementation that fits the interface/contract. Changes to the interface require a rebuild; changes to the implementation require rebuilds only for the runtime, so the type checker can ignore those, where the full signature/interface is not inferred from the implementation (this probably already happens with --build anyway, but on the inter package level it would further reduce the number of modules/exports to be revisited). isolatedDeclarations can be used as an option to minimize the latter on a per package.json/tsconfig.json basis.

This would bring things quite a bit closer to OP's request, wouldn't it? The remaining difference would be "one build followed by checks" versus "check first build later", but the overall amount of compilation would essentially be the same.

[Edit: As @RyanCavanaugh points out later in this thread, with accepting this as being equivalent, one arrives at #25376]

This assumes monorepo builds across build tools / package managers can work off of package.json information alone, apart from maybe topological information (dependency graphs / build graphs). And it still might get a bit hairy, because tools like pnpm and turbo have special configs/fields that influence installation/build order and concurrency, which in turn could influence which typings are (or should be) available at a particular build step (on the other hand, turbo probably already works with TS as is and the addition of this new feature won't have a negative impact / can be omitted in this scenario anyway).

Note: Haven't thought about monorepos which use references that aren't split 1:1 into packages, so maybe a composite enum won't work, but reading this, one probably gets the idea.

Anyway, just dropping ideas for inspiration.

[RyanCavanaugh]

This is a duplicate of #25376, then. Everything you said there (e.g. don't rebuild downstream projects if .d.ts didn't change) is already true of tsc -b