--watch compilation with no change takes 3 seconds

[zen0wu]

I have a project that's fairly small (167 project TS files, ~8000 lines of code, and 987 total TS files including libraries).

When I work on one of the top modules (I use referenced projects and have 5 modules, the top one is a webserver module that depends on a bunch of heavy TS libraries like nestjs), even though I just save the file as it is (or touch webserver/x.ts), the triggered watch compilation takes about 4 seconds to finish, which is not very efficient to work with.

Results of tsc -b tsconfig.json -w --diagnostics showing most of the time is in parsing and binding.

[8:23:06] File change detected. Starting incremental compilation...

Files:           987
Lines:        422816
Nodes:       1248135
Identifiers:  453635
Symbols:      273809
Types:            73
Memory used: 783590K
I/O read:      0.44s
I/O write:     0.01s
Parse time:    2.95s
Bind time:     0.55s
Check time:    0.00s
Emit time:     0.00s
Total time:    3.50s
[8:23:10] Found 0 errors. Watching for file changes.

I did some simple profiling of tsc.js with the process above, and it appears the compiler takes significant amount of time scanning all the modules, but my gut feeling is all these files/dependencies can be cached.

I wonder if there's any obvious mistakes I'm making, or it's an item on the roadmap since it'd be a big improvement if this can be better.

TypeScript Version: 3.5.2

Search Terms:

Expected behavior:
Since no program change happens, it should be nearly instant to finish the compilation.

Actual behavior:
It takes 3-4 seconds to finish the watch compilation.

Playground Link:

Related Issues:

[fatcerberus]

Are you using --incremental? That was introduced in 3.4 and improved in 3.5 so I'm thinking it might help here.

[zen0wu]

Oh, I forgot to mention. Since I'm using project reference, so all my modules have composite: true enabled, which infers incremental.

[zen0wu]

Did another experiment, where I converted the project into a singular project, without project reference. Now touching a file without any change and trigger compilation under -w is faster, like under 1 second.

Update: With singular project, it has to be -p tsconfig.json rather than -b tsconfig.json, otherwise it won't speed up. But of course the issue is, without -b, the startup of compilation is pretty slow.

[sheetalkamat]

@ZenoZen For scalability we do not yet cache module resolution across compilations in watch mode with --build mode. (consider having too many projects and running out of memory because of that). We do want to investigate about storing such information for the currently edited/changed programs and that might help your situation.
cc: @RyanCavanaugh @DanielRosenwasser

[zen0wu]

@sheetalkamat Thanks for replying!

So it means that recompilation with either -w or --incremental would scan all the files, that's intentional, right? It seems that there is info about scanned modules/files in .tsbuildinfo file, we're just not using them?

Is there anything I can do for now to make it better, other than switching back to non-referenced project structure?

[RyanCavanaugh]

@sheetalkamat let's think long-term about possible improvements here

[chyzwar]

It seems that with incremental tsc still parse all files? I would think that if hash is the same there should be no need to parse files ?

In my case incremental makes compilation slower:

{
    "compilerOptions": {
        "module": "commonjs",
        "moduleResolution": "node",
        "target": "es2018",
        "strict": false,
        "sourceMap": true,
        "declaration": true,
        "emitDeclarationOnly": true,
        "allowJs": false,
        "incremental": true,
        "composite": true,
        "esModuleInterop": true,
        "jsx": "react",
        "rootDir": "src",
        "outDir": "types",
        "skipLibCheck": true,
        "tsBuildInfoFile": "types/tsBuildInfo.json",
        "types": [
            "react",
            "jest",
            "prop-types",
        ],
        "lib": [
            "esnext",
            "dom"
        ]
    },
    "include": [
        "src/**/*.ts",
        "src/**/*.tsx",
        "src/**/*.json"
    ],
    "exclude": [
        "src/**/*.test.ts",
        "src/**/*.test.tsx"
    ]
}

Result in

$ tsc --build --diagnostics
Files:           86
Lines:        77126
Nodes:       242645
Identifiers:  90151
Symbols:      48461
Types:         2376
Memory used: 90497K
I/O read:     0.02s
I/O write:    0.00s
Parse time:   1.22s
Bind time:    0.68s
Check time:   0.38s
Emit time:    0.02s
Total time:   2.30s
Done in 2.66s.

After nuking outDir and disabling incremental

{
    "compilerOptions": {
        "module": "commonjs",
        "moduleResolution": "node",
        "target": "es2018",
        "strict": false,
        "sourceMap": true,
        "declaration": true,
        "emitDeclarationOnly": true,
        "allowJs": false,
        "incremental": false,
        "composite": false,
        "esModuleInterop": true,
        "jsx": "react",
        "rootDir": "src",
        "outDir": "types",
        "skipLibCheck": true,
        "types": [
            "react",
            "jest",
            "prop-types"
        ],
        "lib": [
            "esnext",
            "dom"
        ]
    },
    "include": [
        "src/**/*.ts",
        "src/**/*.tsx",
        "src/**/*.json"
    ],
    "exclude": [
        "src/**/*.test.ts",
        "src/**/*.test.tsx"
    ]
}

$ tsc --build --diagnostics
Files:           86
Lines:        77126
Nodes:       242645
Identifiers:  90151
Symbols:      48461
Types:         2382
Memory used: 94655K
I/O read:     0.01s
I/O write:    0.00s
Parse time:   1.04s
Bind time:    0.58s
Check time:   0.32s
Emit time:    0.08s
Total time:   2.03s
Done in 2.36s.

I am using latest version of tsc.

[zen0wu]

@chyzwar Yeah, IMU, --incremental works pretty much the same way as --watch, so they both parse all the files with --build mode.

Right now I actually moved to lerna/yarn workspaces sort of project structure as the codebase grows, but because of this issue, I have to hack how yarn links projects, to have a single tsconfig.json (to avoid using --build mode) and re-link all node_modules/@my-project/<module> to the outDir/<module>. It's extremely hacky and has a few downside...

[garybernhardt]

I think that I'm seeing this same issue. My simplified test configs as a gist: https://gist.github.com/garybernhardt/5c5aeb9d1bb361d28b34a1f6c3aa0a3b. Our app is closed-source, so I can't post it for repro. I did try to make a minimal repro app, but the lack of actual source to compile made the builds so fast that I couldn't measure the incremental build slowdown.

Here are some data points about build times with various configurations. This is on 3.7.0-dev.20191018. I'm measuring tsc's self-reported timestamps, seven trials in each case, from "Starting incremental compilation..." until "Found 0 errors." In each case I'm adding or removing a single console.log(1) to a leaf module in the module graph.

1. Without project references or "composite": true
   • (This uses the configs above, but collapsed into a single tsconfig.json with no "composite" or "references".)
   • $(npm bin)/tsc --watch --preserveWatchOutput
   • Timings (s): 1, 0, 1, 0, 0, 0, 0.
2. With "composite": true in common's tsconfig
   • (This uses the exact configs above.)
   • $(npm bin)/tsc -b src/server --watch --preserveWatchOutput
   • Timings (s): 1, 1, 1, 1, 1, 2, 1
3. With "composite": true in both tsconfigs ("server" and "common")
   • (This uses the exact configs above, but with "composite": true added to src/server/tsconfig.json.)
   • $(npm bin)/tsc -b src/server --watch --preserveWatchOutput
   • Timings (s): 6, 5, 5, 4, 6, 6, 7

The last one shows this becoming prohibitively slow, and is similar to the timings that I saw when building our full system (client, server, and common). Our project is a bit bigger than @ZenoZen's, but definitely not large (193 TS files, 16,924 lines of TS). Slowdown seems to be ~20x on identical source code.

[garybernhardt]

Is any work on this planned?

[timocov]

Looks like I've faced the same issue.

The build time for non-composite project in watch mode for the only 1 change is fast (around 2-3 seconds), but for the same project in --build mode it takes 1-2 minutes. Looks like tsc in --build mode recompiles the whole sub-project (why it doesn't use the mode it uses for non-composite projects to watch changes quickly?) even if the only 1 file was changed.

Our project is closed-source, but let me know if I can provide any to help to fix it. Composite project is really great feature and we're going to move our codebase to it, but the issue is blocker for that (we can't wait for 1 minutes for the only 1 file change).

consider having too many projects and running out of memory because of that

@sheetalkamat what's the difference between the same project without --build mode? Could it be crashed with that? I meant is this really problem which should be solved? I didn't seen a lot of projects moved to composite projects, but what's I've seen previously used "simple" mode and nothing is crashed. Please bear with me, I might miss something.

Without --build mode, watch, the only 1 file was changed:

Files:                         3940
Lines:                       453223
Nodes:                      1964511
Identifiers:                 654168
Symbols:                     306873
Types:                         1150
Memory used:               1181739K
Assignability cache size:       378
Identity cache size:              0
Subtype cache size:              20
Strict subtype cache size:        4
I/O Read time:                0.00s
Parse time:                   0.01s
Program time:                 0.12s
Bind time:                    0.02s
Check time:                   0.17s
transformTime time:           0.02s
commentTime time:             0.01s
printTime time:               0.80s
Emit time:                    0.80s
I/O Write time:               0.01s
Total time:                   1.12s

--build mode, watch, the same file was changed:

Files:                         3940
Lines:                       453358
Nodes:                      1962640
Identifiers:                 653221
Symbols:                     854739
Types:                       341902
Memory used:               1141882K
Assignability cache size:    217546
Identity cache size:          11335
Subtype cache size:           16806
Strict subtype cache size:    27107
I/O Read time:                0.66s
Parse time:                   4.37s
Program time:                23.16s
Bind time:                    2.73s
Check time:                  51.00s
transformTime time:           3.22s
commentTime time:             0.01s
printTime time:               1.76s
Emit time:                    1.76s
I/O Write time:               0.02s
Total time:                  78.66s

[timocov]

Hi @RyanCavanaugh, may I ask to say what kind of proposal we expect here? How to fix slowness of compilation or something else? Maybe somebody from this thread needs to provide more information which might help to choose better solution?

[GarryOne]

In my case the issue was fixed after upgrading to the the version 4.0.3 from 3.9.7

[DanielRosenwasser]

If people here would be willing to try out tomorrow's nightly build, we'd appreciate the feedback!