Incremental build performance dropped

[use-strict]

Hi,

I noticed that our incremental builds started to be considerably slower than they used to, so I ran webpack with --profile --progress and older versions of ts-loader to see if there are differences. It seems that v0.5.2 is twice as fast as the latest version. I'm unsure how to interpret the profiling data and the webpack source code is a complete mess, so it doesn't help much in that regard.

Here are the stats:

// ts-loader v0.5.2
379ms build modules   
19ms seal
39ms optimize
60ms hashing
35ms create chunk assets
2ms additional chunk assets 
0ms optimize chunk assets 
0ms optimize assets 
99ms emit
Hash: fdd096aa48ded051dda3
Version: webpack 1.12.2
Time: 639ms

// ts-loader v0.5.6
357ms build modules   
19ms seal
50ms optimize
51ms hashing
40ms create chunk assets
4ms additional chunk assets 
0ms optimize chunk assets 
826ms optimize assets
78ms emit
Hash: fdd096aa48ded051dda3
Version: webpack 1.12.2
Time: 1429ms

It looks like there is a lot of time spent in "optimize assets". I don't know what's causing this, but I have a feeling that ts-loader is using some extra dependencies now which may have the side-effect of enabling some webpack optimizations that are meant for production, not development.

[jbrantly]

Hmm... interesting. Thanks for reporting, I'll take a look.

[use-strict]

I did a little profiling with node-inspector and it seems that time is spent calling languageService.getSemanticDiagnostics(). Can't figure out from the history, if it's called more often than it's supposed to be or if it's a side-effect of some other change. Any ideas?

[use-strict]

Could be this commit: dc53ef6

Before it was checking for "visited" modules.

EDIT: it runs it for every TS file in the project and even lib.d.ts

[jbrantly]

Yea, that was my thought as well. It was put in place to fix this but I think #68 is the way to go and there is a decent chance that it will be more performant.

[use-strict]

Yes, seems a reasonable assumption. Also, in my case there were only ~20 files or so, because the rest are still ye olde' JS files, but in a larger project, this could sum up considerably to the overall incremental build time.

[SonofNun15]

We ran into this very recently. When bundling very large projects with ts-loader, it takes a very long time, which makes iterative development very difficult. tsc doesn't take nearly so long so we've even considered using a gulp task or npm script to pipe tsc first and then webpack the js files just to speed things up. If this can be solved in ts-loader we would be happy! 😀

[jbrantly]

@use-strict I'm working on reproducing locally. I'm seeing something similar to you, but I just wanted to confirm whether or not the numbers you posted were for the initial build or for a change in watch mode. Here's some numbers that I'm seeing:

// initial build
2589ms build modules    
14ms seal
5ms optimize 
14ms hashing
19ms create chunk assets
4ms additional chunk assets 
0ms optimize chunk assets 
784ms optimize assets
7ms emit 
Hash: 58c08445fb896261fe7e
Version: webpack 1.12.2
Time: 3460ms

// after making a change while in watch mode
85ms build modules    
4ms seal 
5ms optimize 
10ms hashing
11ms create chunk assets
4ms additional chunk assets 
0ms optimize chunk assets 
805ms optimize assets
11ms emit
Hash: 1b9954af6b2aacc4d866
Version: webpack 1.12.2
Time: 1023ms

In other words, for the initial build both "build modules" and "optimize assets" takes a long time, but a change in watch mode results in a small "build modules" but still a large "optimize assets"

[jbrantly]

Alright, so after playing with this a little bit more my hopes at it being a quick-fix are gone. I was able to get the ~800ms "optimize assets" down to ~500ms in my test. That time is spent generating errors, and I'm not sure how much more I can reduce that (but I haven't given up yet).

However, I did want to point out there are multiple scenarios & timings to take into account here. There is the "first build" vs an "incremental build" in watch mode, and there is the time to emit output for the modules and time to generate errors. As things stand right now the first build scenario performance is pretty abysmal. In my test it takes 2600ms to generate modules for the first build, but only 85ms to generate modules for an incremental build. At my job we have a project that takes around 20s to generate the first build but less than a second for incremental builds. The point here is that if you're not already using watch mode I highly suggest you do so. That said, I think there are definitely performance gains to be made for the first build and I will look deeper into that.

Regarding the time to generate errors, it is a constant for both the first build and incremental builds, but it's also not a great constant.

So to recap:

Scenario	Action	Time	Notes
First Build	Emit	2600ms	This is bad and I think it can be better
First Build	Gen. Errors	500ms	This is also bad but I'm not sure how much I can fix it
Second Build	Emit	85ms	This is working as intended
Second Build	Gen. Errors	500ms	Same as above

[jbrantly]

Also, I wanted to point out that in my test running tsc takes about 1000ms. So if I can get the first build emit time for my test down to 500ms I would consider that a huge win.

[use-strict]

@jbrantly , initial build time is irrelevant. It's expected to take a long time and it's not a problem as long as I don't have to kill the watch and restart it all the time.

Now some remarks and questions:

• v0.5.2 of ts-loader did not have this overhead, provided it was only checking changed files for errors.
• Checking ALL of the files should scale linearly with project size, which is very bad for us (and for @SonofNun15 apparently)
• Why do we need to check files that have not changed for errors? I know you probably had a good reason for that change, but I need to understand it in the hopes that I can try to provide a better solution. It's also not obvious for me from the linked test.
• Wouldn't the fix with program.getSemanticDiagnostics() work?

[jbrantly]

Checking ALL of the files should scale linearly with project size

You're probably right, but this is something I want to look into more. I don't know if maybe TS has some optimizations in this area that I'm simply missing out on because I'm doing something silly.

Why do we need to check files that have not changed for errors

Consider app.ts -> dep.ts. dep exports something which app uses, but dep changes that export in some way which makes app's usage incompatible. If we only check changed files we won't see the error since app didn't change.

Somewhat related is #52 which deals with the dependency tree. If I'm thinking about things correctly, once that's fixed in the scenario above webpack would consider dep to be a dependency of app, which means if dep changed app would get rebuilt as well. This is bad for timing emit-wise, but could potentially allow going back to the "visited" pattern for diagnostics which might be a net win.

Wouldn't the fix with program.getSemanticDiagnostics() work?

That's what I tried yesterday. It took the time in my test from 800ms to 500ms, so while it was better it was not a magical fix out of the box. But like I mentioned, it's possible I'm doing something silly elsewhere that's causing TS not to be optimized for this case.

FWIW, I'm using this for my test.

[jbrantly]

Just brainstorming potential fixes here: It might be possible to use transpileModule for all emits while still updating the language service (or just a program). Then after compile get diagnostics like we do today. This could reduce the initial build time, but keep the same amount of time to generate diagnostics. One problem with this approach is that it could fail for some applications since it assumes that isolatedModules is turned on.

[use-strict]

I will investigate more today. Looks likeprogram.getSemanticDiagnostics() still iterates through program.getSourceFiles() (https://github.com/Microsoft/TypeScript/blob/master/src/compiler/program.ts#L583-L591)

[use-strict]

I'm still not 100% sure of how the entire process works and there is little documentation to go on, so please just go along me:

Webpack starts with an entry point and recursively includes dependencies, following require calls. At least this is what happens for regular *.js files, without any loaders coming into play.

Now if we add ts-loader, this basically means that every *.ts file spawns a ts-loader instance. Even so, ts-loader itself caches a single typescript instance per project. At this point typescript dependencies are different from webpack dependencies.

Typescript needs to follow any dependencies for type-checking, and that's including any files that are not necessarily referenced at runtime via a require. It's these filesthat webpack has no knowledge of.

After ts-loader is done with a file, a *.js file results and webpack follows again the dependencies that are require calls.

For incremental builds, webpack watches all files in the project, so detecting a change is not a problem. The problem is figuring out which files were actually changed since the last incremental build, which webpack cannot help us with because it have no knowledge of them.

So to sum things up, we need to build the list of changed files ourselves, using only information coming from the compiler API and/or filesystem, maybe using webpack just to get the paths that are being watched. And we need to do this faster than it takes to get all errors from all files in the project. Or, we need to find another way of getting the errors in a cheaper way.

[jbrantly]

Webpack starts with an entry point and recursively includes dependencies, following require calls. At least this is what happens for regular *.js files, without any loaders coming into play.

Yes

Now if we add ts-loader, this basically means that every *.ts file spawns a ts-loader instance.

Roughly. To be pedantic the "loader" function is called for each .ts file.

Even so, ts-loader itself caches a single typescript instance per project.

Yes. (per the "instance" loader option and per compiler instance)

At this point typescript dependencies are different from webpack dependencies.
Typescript needs to follow any dependencies for type-checking, and that's including any files that are not necessarily referenced at runtime via a require. It's these filesthat webpack has no knowledge of.

Right. Simple case would be a .d.ts file which webpack never runs through the loader or attempts to bundle.

After ts-loader is done with a file, a *.js file results and webpack follows again the dependencies that are require calls.

Yup. By the time the second file is put through the loader TypeScript generally has all of the files for the project loaded. This is actually kind of a bad thing though for various reasons.

For incremental builds, webpack watches all files in the project, so detecting a change is not a problem.

Well, to be clear, webpack watches all files that were put through the loader or explicitly added as a dependency using the addDependency API. This can be (and currently is) different from what TypeScript has loaded. Right now we manually add all .d.ts files as dependencies so that they're watched by webpack.

The problem is figuring out which files were actually changed since the last incremental build, which webpack cannot help us with because it have no knowledge of them.
So to sum things up, we need to build the list of changed files ourselves, using only information coming from the compiler API and/or filesystem, maybe using webpack just to get the paths that are being watched.

I'm not sure this is a problem. Webpack can tell us which files were actually changed. For instance it does so today for .d.ts files. The problem is detecting errors in files that were not actually changed but depended on files that were. And webpack could possibly help us there if we instruct it to do so by explicitly adding the deep tree of dependencies that TypeScript knows about to each file.

In other words, instead of calling a blanket getDiagnostics on the entire program, we could potentially build a list of files that changed concatenated with a list of files that depended on those files and call getDiagnostics on those.

As an example, suppose app.ts -> dep.ts.

If dep.ts changes, we need to call getDiagnostics on both app.ts and dep.ts even though only dep.ts changed. However, if app.ts changes then we only need to call getDiagnostics on app.ts. This helps the best-case scenario but doesn't address the worst-case. If you change a file that nothing depends on it will be very fast. But if you change a file that everything depends on then it'll be the same performance as today.

A downside to this approach is that this list of files will be pushed through the loader again which can increase the "build modules" time, but we might be able to put in some optimizations for that.

[SonofNun15]

When you refer to using "watch", do you mean webpack --watch or webpack-dev-server? Do we need to use the in memory bundling of the webpack dev server to see the benefit?

[SonofNun15]

We are still seeing long load times even when using both --watch and webpack-dev-server. I think it is probably because all of our files are linked. One of our projects is a library, so all of the files in the project are linked in a hierarchy:

Components
|- Component1
|  |- Component1_1
|  |- Component1_2
|- Component2

etc, etc.
Each file imports its children and then exports them so that you can reference any component off of the components object. (via import { components } from 'components'; console.log(components.component1.component1_1);)

[SonofNun15]

From what you are saying, my understanding is that this forces ts-loader to recompile the whole project tree whenever one file changes because of the dependencies?

[use-strict]

@SonofNun15 , both webpack --watch and webpack-dev-server do the same thing in regards to compilation. The difference is only in output. The dev server uses an in-memory file system and serves the bundles via http along with some runtime code used for hotloading. The regular webpack just writes them to disk. Both of them stay in memory and watch files for changes. Performance should be nearly identical.

I am actually using a forked dev-server which writes files to disk and only use HTTP to get the stats (errors and warnings) for integrating them with the IDE. So it's basically the same thing.

[SonofNun15]

OK, I see. So webpack --watch keeps the modules loaded in memory, but I assume it does have to flush the whole file to disc on any change. This should be relatively quick given we are talking about KB - MB size files not GB. 😀

[use-strict]

Yes it is very quick. One other thing you should keep an eye on, and I don't know what you use in your project, isdevtool: 'source-map' which has to regenerate the source map for the entire bundle on the smallest change. That also increases build times linearly with project size, so ideally you should be using eval-source-map or something faster.

[jbrantly]

@SonofNun15 When talking about --watch, you're still going to incur the initial build time but then subsequent builds should be a lot faster. I would be interested in hearing your numbers for the initial build and then for a change.

From what you are saying, my understanding is that this forces ts-loader to recompile the whole project tree whenever one file changes because of the dependencies?

Kind of, not exactly. ts-loader does get diagnostics from the entire program for every change, but that isn't the fault of how you've got your project structured.

[SonofNun15]

Current run time numbers are:
Initial load time: ~30 seconds
Reload time (webpack --watch): ~14 seconds

Bundle size is 1.85 MB

We were using dev-tool: "inline-source-map". I removed it before recording the above numbers.

[SonofNun15]

This is without minification, which reduces the bundle size to ~638KB

[use-strict]

@jbrantly , I opened this issue: microsoft/TypeScript#5192

[jbrantly]

Interesting, thanks. Good to know they're aware of the problem of discovering deep dependencies. I still think there is a good possibility we can fix this now, but long term using an official API would clean things up.

I haven't had the time to really explore this yet due to other obligations but I definitely do want to fix.

[jbrantly]

FWIW, I've been thinking on this a lot and I believe I know a way to improve the initial build time. I'm planning on trying it out sometime this week. I realize this issue is more about incremental build time but initial build time is important too.

I still have ideas on the incremental build time but they're not as fleshed out. I'm also planning on exploring that more.

[use-strict]

It's good to optimize both if possible, although the main bottleneck for the initial build time is probably not ts-loader, but webpack itself.

[jbrantly]

Actually there is quite a bit of bottleneck in ts-loader right now. Using tsc to generate js and then just webpacking the js is significantly faster than using ts-loader for large projects. The issue (I believe) is that we are forcing TypeScript to re-synchronize everything pretty much for each file when we should only need to do it once in the common case. That's what I'm planning to address.

The idea for improving incremental build time is somewhat dependent on the resulting behavior of the above, so that's why I want to investigate that afterward. But the basic idea hasn't changed: we need to somehow track dependencies and then only get diagnostics for the changed files and their deep dependents. The trick is that "somehow" 😁

[raybooysen]

As a potentially distracting aside, I recently switched from awesome-tsloader to ts-loader and for the same project, my build increased by a minute using ts-loader for an initial build.

[jbrantly]

@raybooysen Still a long time 😦 Hopefully I can cut it down a lot more. Do you happen to have a loader that executes before ts-loader?

[raybooysen]

Yes we have a few loaders. I wish webpack would output timings for this.

[raybooysen]

Actually saying that, I need to confirm if the loaders run before ts-loader. I think not. But this is a full build, I don't have a breakdown of how much time each loader is taking.

[Karabur]

Can we add an option to skip error checking at after-compile?
Since if someone uses incremental build, it is most likely it is used in development, and IDE already showing all those errors. So skip them in webpack output with the benefit of HUGE speed increase worth it, and does not looks hard to implement.
I've disabled after-compile hook in ts-loader and got an improvement from 6 sec to 15ms in incremental build.

[raybooysen]

Pop up a PR. :)

[use-strict]

@Karabur , the IDE probably shows you errors in the file being currently edited, it won't show you errors in other files, caused by changes in the current file. In other words, it only helps with syntactic errors, not semantic errors. You will get those only by incremental build/ recompilation of the entire project. Because of this, I'm inclined to think that skipping error reporting provides little to no benefit. It may even prove detrimental if you discover errors later, because it will take you more time to figure out when and where you did something wrong.

[Karabur]

@raybooysen ok, will try to make a proper one.
@use-strict skipping error reporting provides no benefit, indeed. But getting compilation speedup like that has. And it is important when you have hot-reloading for example. And errors not hidden completely, you can set up your environment to easily see them at any time you would like to.

I am not sure about all IDE's but my (WS) shows errors for all files, not only for which I am currently edit. I dont need webpack output at all and have no benefit from loader forcing me to to wait 6-15 seconds on every single change to see app updated, but drawback is huge. Anyway it is matter of workflow, and since impact could be significant, it is always better an option to choose.

[use-strict]

IMHO, in the end, it still comes down to the performance of the incremental compilation. Anything larger than a few seconds is not acceptable, for me anyway. Skipping error reporting is not the right solution here. Getting the errors in an efficient way is.

I'm not sure in your case if those 6-15 seconds are just typescript compilation or overall time, including time spent by webpack. The usual bottleneck is dependency resolving, so there may be other things causing the large build time.

On a side-node, something to watch out for: WS has its own "language service" / compilation logic. It doesn't use the API exposed by typescript and they said it will never do so, for commercial reasons. Although rare, you might find yourself with compilation errors when the IDE says everything is fine and viceversa. WS10 at least was full of this kind of bugs and it was one of the reasons why I moved away from it.

[jbrantly]

@Karabur Have you tried using the transpileOnly option?

[Karabur]

@use-strict of course it is workaround, not the solution. 6-14 is error processing step only. so, having all incremental time about 7 seconds I have like 1 second without. For relativeliy small project. Other people reported more time on that step.
Your info about WS is outdated and incorrect. You can use exact same TS compiler as ts-loader and have 100% of it's errors. But it is off-topic here, and other editors could have full type checking support as well (why they shouldn't?)
@jbrantly just tried it:
without any options: total build time: 10Sec (9 seconds overhead)
with transpileOnly: 3~4 sec (2-3 seconds overhead)
with errors disabled: 1 sec
transpileOnly seems to give good boost, and, since with transpileOnly there is also no errors printed, I think it is safe and correct to skip after-compile step completely if it is set to true

[Karabur]

I've checked code in more details and with transpileOnly options error gather is never called at all. Looks like things was messed up in my test runs, so that 1s time is caused by something else. I'll try to do more isolated and clean performance testing. So far transpileOnly looks like best option.

[jbrantly]

Right. With transpileOnly the whole after-compile step is completely skipped. transpileOnly basically is the option to skip error checking (there is some minor error checking that goes on but I think it's only syntactic).

[jbrantly]

I should note (probably in the documentation somewhere) that transpileOnly does have slightly different output behavior:

From here

imports\exports are not elided just because they cannot be resolved (since we have code for only one file)
const enums are treated as regular enums (even for locally defined const enums)
files are always emitted as modules even if they don't have top level export \ import \ require

[panKt]

Any news or workarounds to make latest version of ts-loader as fast as 0.5.2?

[jbrantly]

@panKt Not yet. I believe initial build time is actually faster now than in 0.5.2, but incremental build time still has the issue outlined here.

[NikGovorov]

@panKt, As a workaround you could use transpileOnly during development.

[waeljammal]

I am having this issue too :( incremental compile gets slower and slower as the project gets bigger, I've resorted to using transpileOnly for now but it's bad because I keep having to remember to toggle it to check for errors :(

[raybooysen]

Is it getting slower in relation to how much code you're adding? This
would be expected then.

On 20 January 2016 at 09:40, Wael notifications@github.com wrote:

I am having this issue too :( incremental compile gets slower and slower
as the project gets bigger, I've resorted to using transpileOnly for now
but it's bad because I keep having to remember to toggle it to check for
errors :(

—
Reply to this email directly or view it on GitHub
#78 (comment)
.

[spock123]

Had the same issue, setting "sourceMap": false in tsconfig.json helped tremendously (much more than expected).

[kenjiru]

Setting "sourceMap": false doesn't change much in my case. But setting transpileOnly brings down the compile time from 2.5s to 700ms. This might now seem much, but it is actually 3 times faster.

[jklmli]

awesome-typescript-loader stores an internal dependency graph. Would this help at all?

FWIW, both ts-loader and awesome-typescript-loader seem to have an issue with optimize-assets taking a long time :(