Support "medium-sized" projects

[sophiajt]

Listening to @nycdotnet has me fired up to tackle this one. Thanks, Steve. (btw, you can check out his good interview here: http://www.dotnetrocks.com/default.aspx?showNum=1149)

The proposal here was first started in times prehistoric (even before #11), when dinosaurs walked the scorched earth. While nothing in this proposal is novel, per se, I believe it's high time we tackled the issue. Steve's own proposal is #3394.

The Problem

Currently, in TypeScript, it's rather easy to start and get going, and we're making it easier with each day (with the help of things like #2338 and the work on System.js). This is wonderful. But there is a bit of a hurdle as project size grows. We currently have a mental model that goes something like this:

• Small-sized projects: use tsconfig.json, keep most of your source in the current directory
• Large-sized projects: use custom builds, put source where you need it

For small-sized projects, tsconfig.json gives you an easy-to-setup way of getting going with any of the editors in a cross platform way. For large-scale projects, you will likely end up switching to build systems because of the varied requirements of large-scale projects, and the end result will be something that works for your scenarios but is difficult to tool because it's far too difficult to tool the variety of build systems and options.

Steve, in his interview, points out that this isn't quite the right model of the world, and I tend to agree with him. Instead, there are three sizes of project:

• Small-sized projects: use tsconfig.json, keep most of your source in the current directory
• Medium-sized projects: those with standard builds and shared components
• Large-sized projects: use custom builds, put source where you need it

As you scale in size of project, Steve argues, you need to be able to scale through the medium step, or tool support falls off too quickly.

Proposal

To solve this, I propose we support "medium-sized" projects. These projects have standard build steps that could be described in tsconfig.json today, with the exception that the project is built from multiple components. The hypothesis here is that there are quite a number of projects at this level that could be well-served by this support.

Goals

Provide an easy-to-use experience for developers creating "medium-sized" projects for both command-line compilation and when working with these projects in an IDE.

Non-goals

This proposal does not include optional compilation, or any steps outside of what the compiler handles today. This proposal also does not cover bundling or packaging, which will be handled in a separate proposal. In short, as mentioned in the name, this proposal covers only the 'medium-sized' projects and not the needs of those at large scales.

Design

To support medium-sized projects, we focus on the use case of one tsconfig.json referencing another.

Example tsconfig.json of today:

{
    "compilerOptions": {
        "module": "commonjs",
        "noImplicitAny": true,
        "sourceMap": true
    },
    "files": [
        "core.ts",
        "sys.ts"
    ]
}

Proposed tsconfig.json 'dependencies' section:

{
    "compilerOptions": {
        "module": "commonjs",
        "noImplicitAny": true,
        "sourceMap": true
    },
    "dependencies": [
        "../common", 
        "../util"
    ],
    "files": [
        "core.ts",
        "sys.ts"
    ]
}

Dependencies point to either:

• A directory, where a tsconfig.json can be found
• A tsconfig.json directly

Dependencies are hierarchical. To edit the full project, you need to open the correct directory that contains the root tsconfig.json. This implies that dependencies can't be cyclic. While it may be possible to handle cyclic dependencies in some cases, other cases, namely those with types that have circular dependencies, it may not be possible to do a full resolution.

How it works

Dependencies are built first, in the order they are listed in the 'dependencies' section. If a dependency fails to build, the compiler will exit with an error and not continue to build the rest of the project.

As each dependency completes, a '.d.ts' file representing the outputs is made available to the current build. Once all dependencies complete, the current project is built.

If the user specifies a subdirectory as a dependency, and also implies its compilation by not providing a 'files' section, the dependency is compiled during dependency compilation and is also removed from the compilation of the current project.

The language service can see into each dependency. Because each dependency will be driven off its own tsconfig.json, this may mean that multiple language service instances would need to be created. The end result would be a coordinated language service that was capable of refactoring, code navigation, find all references, etc across dependencies.

Limitations

Adding a directory as a dependency that has no tsconfig.json is considered an error.

Outputs of dependencies are assumed to be self-contained and separate from the current project. This implies that you can't concatenate the output .js of a dependency with the current project via tsconfig.json. External tools, of course, can provide this functionality.

As mentioned earlier, circular dependencies are considered an error. In the simple case:

A - B
\ C

A is the 'current project' and depends on two dependencies: B and C. If B and C do not themselves have dependencies, this case is trivial. If C depends on B, B is made available to C. This is not considered to be circular. If, however, B depends on A, this is considered circular and would be an error.

If, in the example, B depends on C and C is self-contained, this would not be considered a cycle. In this case, the compilation order would be C, B, A, which follows the logic we have for ///ref.

Optional optimizations/improvements

If a dependency does not to be rebuilt, then its build step is skipped and the '.d.ts' representation from the previous build is reused. This could be extended to handle if the compilation of a dependencies has built dependencies that will show up later in the 'dependencies' list of the current project (as happened in the example given in the Limitations section).

Rather than treating directories passed as dependencies that do not have a tsconfig.json as error cases, we could optionally apply default 'files' and the settings of the current project to that dependency.

[nycdotnet]

Oh my goodness!

[unional]

👍

[kungfusheep]

Yep! This makes perfect sense for the use-cases you've provided. Providing the tools we use with a better view on how our code is intended to be consumed is definitely the right thing to do. Every time I F12 into a .d.ts file by mistake I feel like strangling a kitten!

[nycdotnet]

Jonathan,

Thank you so much for the kind feedback and for deciding to take this on. TypeScript is such a great tool and this functionality will help many people who want to componentize their medium-sized codebases that can't justify the inefficiency mandated by enormous projects that rely on strict division of concerns (for example the Azure portals or project Monacos of the world with > 100kloc and many independent teams). In other words, this will really help "regular people". Also, others have already proposed stuff for this, for example @NoelAbrahams (#2180), and others so I can't claim originality here. This is just something I've been needing for a while.

I think that your proposal is excellent. The only shortcoming that I see versus my proposal (#3394), which I have now closed, is the lack of a fallback mechanism for references.

Consider the following real-world scenario which I detailed here: #3394 (comment)

I have a TypeScript project grunt-ts that depends on a different project csproj2ts. Hardly anyone who works on grunt-ts will also want to work on csproj2ts as it has a very limited scope of functionality. However, for someone like me - It'd be great to be able to work on both projects simultaneously and do refactoring/go to definition/find all references across them.

When I made my proposal, I suggested that the dependencies object be an object literal with named fallbacks. A version more in keeping with your proposal would be:

"dependencies": {
   "csproj2ts": ["../csproj2ts","node_modules/csproj2ts/csproj2ts.d.ts"],
   "SomeRequiredLibrary": "../SomeRequiredLibraryWithNoFallback"
}

To simplify it to still be an array, I suggest the following alternate implementation of a future hypothetical dependencies section of the grunt-ts tsconfig.json file:

"dependencies": [
   ["../csproj2ts","node_modules/csproj2ts/csproj2ts.d.ts"],
   "../SomeRequiredLibraryWithNoFallback"
]

The resolution rule for each array-type item in dependencies would be: the first item that is found in each is the one that is used, and the rest are ignored. String-type items are handled just as Jonathan's proposal states.

This is a very slightly more complicated-to-implement solution, however it gives the developer (and library authors) far greater flexibility. For developers who do not need to develop on csproj2ts (and therefore do not have a ../csproj2ts/tsconfig.json file), the dependency will just be a definition file that gets added to the compilation context. For developers who do have a ../csproj2ts/tsconfig.json file, the proposal will work exactly as you've described above.

In the above example, the "../SomeRequiredLibraryWithNoFallback" would be required to be there just like in your existing proposal, and its absence would be a compiler error.

Thank you so much for considering this.

[mhegazy]

There are two probelms here that we need to break apart, there is build, and there is language service support.

For Build, I do not think tsconfig is the right place for this. this is clearly a build system issue. with this proposal the typescript compiler needs to be in the businesses of:

• figuring out dependency
• up-to-date checks fro asserts
• configuration management (release vs. debug)

These are all clearly the responsibility of build systems; these are hard problems and there are already tools that do that, e.g. MSBuild, grunt, gulp, etc..
Once tsconfig and tsc become the build driver you would want it to use all CPUs to build unrelated subtrees, or have post and pre build commands for each project, and possibly build other projects as well. again, i think there are build tools out there that are good at what they do, and no need for us to recreate that.

For Language Service,
I think it is fine for tooling to know about multiple tsconfig files and can infer from that and help you more, but that should not affect your build. i would consider something like:

"files" : [
    "file1.ts",
    {
        "path": "../projectB/out/projectB.d.ts",
         "sourceProject": "../projectB/"
     }
]

Where tsc will only look at "path" but tools can look at other information and try to be as helpful as they can.

I acknowledge the existence of the problem but do not think lumping build and tooling is the correct solution. tsconfig.json should remain a configuration bag (i.e. a json alternative to response files) an not become a build system. one tsconfig.json represents a single tsc invocation. tsc should remain as a single project compiler.

MSBuild projects in VS are an example of using a build system to build IDE features and now ppl are not happy with it because it is too big.

[nycdotnet]

Thanks for your reply, Mohamed. Let me restate to see if I understand:

• You think that the task of coordinating multi-project builds should remain the domain of dedicated build tools.
• You think that there may be something to this suggestion for the TypeScript language service.
• You think that running tsc --project on this tsconfig.json would be the same as running tsc file1.ts ../project/out/project.d.ts. However, opening such a project in VS or another editor with the TypeScript language service would allow "go to definition" to bring the developer to the actual TypeScript file where the feature was defined (rather to the definition in projectB.d.ts)

Do I have that right?

If so, I think this is very fair. In my original proposal (#3394), I did say that my idea was incomplete because it didn't include the step of copying the emitted results from where they'd be output in the referenced library to where the referencing library would expect them at runtime. I think you're saying "why go halfway down a road for building when what is really needed is the language service support".

To change the data a bit in your example, would you be willing to support something like this?

"files" : [
    "file1.ts",
    {
        "path": "externalLibraries/projectB.d.ts",
         "sourceProject": "../projectB/"
     }
]

The assumption is that the current project would ship with a definition for projectB that would be used by default, but if the actual source for projectB is available, the actual source would be used instead.

[mhegazy]

@nycdotnet you summed it up right; i would like to create a system that is loosely coupled and allows mixing and matching diffrent build tools with diffrent IDEs but still get a great design time experience.

[nycdotnet]

Sounds great!

[frankwallis]

I agree with @mhegazy and actually I think it is important for TypeScript to stop thinking of itself as a 'compiler' and start thinking of itself as a 'type-checker' and 'transpiler'. Now there is single-file transpilation support I don't see any reason for compiled JavaScript files to be created except for at runtime/bundling. I also don't see why it is necessary to generate the external reference definitions during type-checking when the actual typescript source is available.

File and package resolution is the responsibility of the build system you are using (browserify, systemjs, webpack etc), so for the tooling to work the language service needs to be able to resolve files in the same way as whatever build system/platform you are using. This either means implementing a custom LanguageServicesHost for each build system or providing a tool for each one which generates the correct mapping entries in tsconfig.json. Either of these is acceptable.

@nycdotnet I think your use case for multiple fallback paths would be better handled using npm link ../csproj2ts?

[dbaeumer]

I agree with @mhegazy that we should keep build separate from the compiler/transpiler. I do like to include the tsconfig -> tsconfig dependency in the files section assuming that if the section doesn't list any *.ts files it still scans for them. E.g.

"files" : [
{
"path": "externalLibraries/projectB.d.ts",
"sourceProject": "../projectB/"
}
]

Will still include all ts files in the directory and subdirectory containing the tsconfig.json file.

[mhegazy]

@dbaeumer you then want to have it in a different property, correct? currently, if files is defined, it is always used and we ignore the include *.ts part.

[dbaeumer]

@mhegazy not necessarily a different section although it would make things clearer at the end. All I want to avoid is to be forced to list all files if I use a tsconfig -> tsconfig dependencies. In the above example I would still like to not list any *.ts files to feed them into the compiler.

[billti]

I think this is much needed. I don't think we can avoid the build question however. That doesn't mean we need to implement a build system along with this proposal, but we should have thought through some good guidance that works in a configuration such as that proposed. It will be non-trivial to get right (and we do need to solve it for Visual Studio).

In the above proposal (where both the .d.ts and the source is referenced), does it detect if the .d.ts is out of date (i.e. needs to be rebuilt)? Do operations like Refactor/Rename work across projects (i.e. updates the name in the referenced project source, not just its .d.ts file which will get overwritten next build)? Does GoToDef take me to the original code in the referenced project (not the middle of a giant .d.ts file for the whole project)? These would seem it imply needing to resolve, and in some cases analyze, the source of the referenced projects, in which case is the .d.ts that useful?

[mhegazy]

The general solution, which we have today, you have a .d.ts as a build output of one project, and then referenced as input in the other project. This works fine for build, so no need to change that.

The problem is the editing scenario. you do not want to go through a generated file while editing. My proposed solution is to provide a "hint" of where the generated .d.ts came from. The language service will then not load the .d.ts, and instead load a "project" from the hint path. this way goto def will take you to the implementation file instead of the .d.ts and similarly errors would work without the need of a compilation.

operations like rename, will "propagate" from one project to another, similarly find references, will do.

[jbrantly]

Today it is completely up to the host (the IDE, whatever) to find the tsconfig.json file, although TS then provides APIs to read and parse it. How would you envision this working if there were multiple tsconfig.json located in a hierarchical fashion? Would the host still be responsible for resolving the initial file but not the others or would the host be responsible for resolving all of the tsconfigs?

Seems like there is a tradeoff there between convenience/convention and flexibility.

[spion]

Wouldn't this start with the ability to build d.ts files as described in #2568 (or at least have relative imports)?

[mhegazy]

@spion i am not sure i see the dependency here. you can have multiple outputs of a project, it does not have yo be a single delectation file. the build system should be able to know that and wire them as inputs to dependent projects.

[spion]

@mhegazy Oops, sorry. Looking at the issue again, it appears that this is more related to the language service. I read the following

• Medium-sized projects: those with standard builds and shared components

and automatically assumed its related to better support for npm/browserify (or webpack) build worfklows where parts of the project are external modules.

AFAIK there is no way to generate .d.ts file(s) for external modules yet? If so, the only way a language service could link projects that import external modules would be to have something like this in tsconfig.json :

{ 
  "provides": "external-module-name"
}

which would inform the LS when the projects is referenced in another tsconfig.json

[mhegazy]

AFAIK there is no way to generate .d.ts file(s) for external modules yet?

I do not think this is true. calling tsc --m --d will generate a declaration file that is an external module itself. resolution logic will try to find a .ts and if not then a .d.ts with the same name,

[nycdotnet]

@spion TypeScript can generate d.ts files for external modules as @mhegazy said, but this results in a 1:1 ratio of definitions to source files which is different from how a library's TypeScript definitions are typically consumed. One way to work around this is this TypeStrong library: https://github.com/TypeStrong/dts-bundle

[spion]

@mhegazy sorry, I meant for "ambient external modules" i.e. if I write external-module-name in TypeScript and import one of its classes from another module:

import {MyClass} from 'external-module-name'

there is no way to make tsc generate the appropriate .d.ts file that declares 'external-module-name'

@nycdotnet I'm aware of dts-bundle and dts-generator but still if the language service is to know about the source of my other project, it should also know what module name it provides to be able to track the imports correctly

[llgcode]

What's the status of this feature? it seems this is an important option for medium size projects. How do you configure a project that have source in different folders with a specific "requirejs" config?

[mhegazy]

@llgcode please take a look at #5039, this should be available in typescript@next.

[octogonz]

As above, module hoisting moves all dependencies to a root node_modules directory. This takes advantage of the fact that node module resolution will always traverse up the directory tree and search through all the node_modules directories until it finds the required module.

Btw there is a downside to this model, that it leads to "phantom dependencies" where a project can import a dependency that was not explicitly declared in its package.json file. When you publish your library, this can cause trouble such as (1) a different version of the dependency getting installed than what was tested/expected, or (2) the dependency missing completely because it was hoisted from an unrelated project that is not installed in this context. PNPM and Rush both have architectural choices intended to protect against phantom dependencies.

[octogonz]

I have a general question about tsc --build: Is the TypeScript compiler seeking to take over the role of orchestrating the build for projects in a monorepo? Normally the toolchain is going to have a whole pipeline of tasks, stuff like:

• preprocessing
• generating localized strings
• converting assets to JavaScript (css, images, etc)
• compiling (type checking / transpiling)
• rolling up the .js files (e.g. webpack)
• rolling up the .d.ts files (e.g. API Extractor)
• post processing including unit tests and documentation generation

Normally a system such as Gulp or Webpack manages this pipeline, and the compiler is just one step in the middle of the chain. Sometimes a secondary tool also runs the build in another way, e.g. Jest+ts-jest for jest --watch.

Is tsc aiming to manage these things itself? And if not, is there a way for a conventional build orchestrator to solve the dependency graph itself, and e.g. repeatedly invoke tsc in each project folder in the right order (after the preprocessing has been updated)?

Or, if the design is to process an entire monorepo in a single pass (whereas today we build each project in a separate NodeJs process), I'm also curious how the other build tasks will participate: For example, will we run webpack on all projects at once? (In the past that led to out-of-memory issues.) Will we lose the ability to exploit multi-process concurrency?

These aren't criticisms BTW. I'm just trying to understand the big picture and intended usage.

[yortus]

@pgonzal right, there are many non-tsc parts to building a real world monorepo. For our lerna monorepo I took the following approach:

• each package in the monorepo optionally defines a prebuild script and/or a postbuild script in its package.json. These contain the non-tsc aspects of the build.
• in the monorepo's package.json, there are these scripts:

      "prebuild": "lerna run prebuild",
      "build": "tsc --build monorepo.tsconfig.json --verbose",
      "postbuild": "lerna run postbuild",
  

• That's it. Running yarn build at the monorepo level runs the prebuild scripts for each package that defines them, then it runs the tsc --build step, then it runs all the postbuild scripts. (By convention in both npm and yarn, executing npm run foo is roughly the same as npm run prefoo && npm run foo && npm run postfoo.)

[octogonz]

How do you handle jest --watch or webpack-dev-server? For example when a source file is modified, do the prebuild/postbuild steps run again?

[borekb]

Does this have any implications on ts-node and related workflows? Some of our helper apps run "directly" from TypeScript, like "start": "ts-node ./src/app.ts" or "start:debug": "node -r ts-node/register --inspect-brk ./src/app.ts".

[yortus]

Reported another issue with build mode at #25355.

[RyanCavanaugh]

Thanks for all the great feedback and investigations so far. I really appreciate everyone who took time to try things out and kick the tires.

@yortus re #3469 (comment)

Great write-up, thanks again for providing this. Your issues in order -

1. Fixed
2. PR up at Handle multiple output .d.ts files changedness correctly #25370
3. Discussing at the logged issue - not immediately apparent what the right fix is, but we'll do something
4. Investigating (below)
5. Logged Infer project references from common monorepo patterns / tools #25376
6. Technically unrelated to --build AFAICT. This is a new assert we added recently; Nathan's investigating

@rosskevin 🎉 for the PR on the learn-a repo! I'm going to merge that into a branch so we can compare and contrast better.

@pgonzal re #3469 (comment)

I have a general question about tsc --build: Is the TypeScript compiler seeking to take over the role of orchestrating the build for projects in a monorepo?

Great question; I want to answer this one very clearly: definitely not.

If you're happy today using tsc to build your project, we want you to be happy tomorrow using tsc -b to build your multi-part project. If you're happy today using gulp to build your project, we want you to be happy tomorrow using gulp to build your multi-part project. We have control over the first scenario, but need tool and plugin authors to help us with the second, which is why even tsc -b is just a thin wrapper over exposed APIs that tool authors can use to help project references play nicely in their build models.

The broader context is that there was fairly robust internal debate over whether tsc -b should even exist, or instead be a separate tool / entry point - building a general-purpose build orchestrator is an enormous task and not one we're signing up for. For our own repo we used tsc with a light task runner framework and now use tsc -b with the same task runner, and I'd expect anyone else migrating to also keep their existing build chain in place with only small tweaks.

@borekb re #3469 (comment)

Does this have any implications on ts-node and related workflows? Some of our helper apps run "directly" from TypeScript

For single-file scripts, which implicitly cannot have project references, there is zero impact.

@EisenbergEffect had some questions in the learn-a repo about cross-project rename and other language service features. The big open question here is just if we'll be able to get this feature in a usable state for 3.0 or not. If so, then cross-project rename will "just work", subject to the caveat that it's obviously impossible for us to conclusively find all downstream projects and update them - this will be a "best effort" based on some heuristics for looking for other projects.

If we don't think cross-project rename is acceptably stable+complete for 3.0, we'll likely block rename operations only when the renamed symbol is in the .d.ts output file of another project - allowing you to do this would be very confusing because the .d.ts file would get updated on a subsequent build of the upstream project after the upstream project had been modified, which means it could easily be days between when you make a local rename and when you realize that the declaring code hadn't actually been updated.

For features like Go to Definition, these are working today in VS Code and will work out-of-the-box in future versions of Visual Studio. These features all require .d.ts.map files to be enabled (turn on declarationMap). There's some per-feature work to light this up, so if you see something not working as expected, do log a bug as we may have missed some cases.

Open problems I'm tracking at this point:

• Cross-project rename - @andy-ms is implementing
• Need to analyze hoisted module setups and understand their implications - on me
• There should be a version of the sample learn-a repo that uses yarn, and another that uses pnpm, and another that uses one of those in hoisted mode

Open questions

• Should we implement logic to read package.jsons to infer project references? Logged Infer project references from common monorepo patterns / tools #25376
• Should .d.ts.map emit be implicitly on for composite projects?

[weswigham]

@RyanCavanaugh to add to

Open problems I'm tracking at this point

We've also mentioned having an incremental output cache, separate from the real project output location, to handle things like updating declarations in the background in the LS (today, changes don't propagate across project boundaries in the editor until you build), stripInternal, and mutating build processes (where our build outputs are mutated in place and so not suitable for LS operations).

[zpdDG4gta8XKpMCd]

sorry for a dumb question, since it's checked in the roadmap, how do i get this feature enabled?

[timfish]

@Aleksey-Bykov you can use it in typescript@next.

I just tried this out on our yarn workspace powered monorepo and it works well.

One thing I did notice was that tsc --build --watch reports errors but then doesn't output anything to say that the build is now fixed. The standard tsc watch mode in 2.9 has started giving an error count and its nice to see a zero there so you know that building has completed.

[zpdDG4gta8XKpMCd]

i have a folder full of *.d.ts and nothing else what am i supposed to do about it:

• make it a project and reference it (tried, didn't work)
• use "include" for it

[RyanCavanaugh]

@timfish that feedback matches some other I've heard; logged #25562

@Aleksey-Bykov #3469 (comment) should help explain some concepts

[zpdDG4gta8XKpMCd]

@RyanCavanaugh it looks like project referecing only works for commonjs and node module resolution, doesn't it?

in you example:

import * as p1 from "@ryancavanaugh/pkg1";
import * as p2 from "@ryancavanaugh/pkg2";

p1.fn();
p2.fn4();

1. what is @ryancavanaugh module, does it has anything to do with how TS resolves modules?
2. is this example supposed to work with AMD (classic module resolution)?
3. is outFile required for definitions to be found?
4. where d.ts files are supposed to be in order for referencing project to find them (can i still use outDir? will TS find them there?)

i have 2 simple projects essentials and common and things in common cannot resolve stuff compiled in essentials:

[RyanCavanaugh]

@Aleksey-Bykov

1. It's just a scoped module name, resolved under the usual node module resolution algorithm
2. You can use project references with any module system including classic, but the example names (scoped modules) aren't very friendly to use outside of node
3. No
4. TypeScript looks for the .d.ts files to be in the place where the referenced project builds them to

If you have a sample repo or something I can diagnose why you're getting that error

[zpdDG4gta8XKpMCd]

@RyanCavanaugh please do
example.zip

[timfish]

@RyanCavanaugh, it also looks like tsc --build --watch doesn't initially output any files until its sees a modification of a source file.

[RyanCavanaugh]

Thread too long (in both time and space); let's pick up the discussion at lucky issue number 100 * 2^8 : #25600