Build Optimizer increasing build time excessively

[alexeagle]

From @filipesilva on November 30, 2017 14:47

In #5618 (comment), @changLiuUNSW reported a 3x increase in build time (from ~3 to ~9 minutes) when using --build-optimizer:

Angular CLI 1.5.2 with Angular 5.0.1

ng build --prod
Time: 530465ms
chunk {0} polyfills.953b2ea9f096cadfdccc.bundle.js (polyfills) 147 kB [initial] [rendered]
chunk {1} main.e2cfa60d1ec303d72a6a.bundle.js (main) 1.77 MB [initial] [rendered]
chunk {2} styles.b49f8f5ee25381105567.bundle.css (styles) 108 kB [initial] [rendered]
chunk {3} inline.2ae12ef65c8a9f6162ba.bundle.js (inline) 1.49 kB [entry] [rendered]

ng build --prod --build-optimizer=false
Time: 163265ms
chunk {0} polyfills.cdff46cd9b1a8d656218.bundle.js (polyfills) 147 kB [initial] [rendered]
chunk {1} main.d1916123bd0e4aa2858f.bundle.js (main) 1.34 MB [initial] [rendered]
chunk {2} styles.d1f7c9f576103c62cb95.bundle.css (styles) 108 kB [initial] [rendered]
chunk {3} vendor.1fbdf23b6f274bb4005e.bundle.js (vendor) 639 kB [initial] [rendered]
chunk {4} inline.58b71d550a47f85197a5.bundle.js (inline) 1.5 kB [entry] [rendered]

angular/devkit#304 has possibly improved this, but needs to be verified.

@changLiuUNSW can you try with Angular CLI 1.5.5 and let us know what's the current build time please?

Likely related to angular/devkit#240.

Copied from original issue: angular/devkit#305

[alexeagle]

From @changLiuUNSW on December 1, 2017 4:19

Thanks @filipesilva. I will have a test tonight and get back to you ASAP.

[alexeagle]

From @changLiuUNSW on December 1, 2017 11:41

$ node -v
v7.5.0

$ ng -v

Angular CLI: 1.5.5
Node: 7.5.0
OS: win32 x64
Angular: 5.0.4
... animations, common, compiler, compiler-cli, core, forms
... http, language-service, platform-browser
... platform-browser-dynamic, router

@angular/cdk: 5.0.0-rc0
@angular/cli: 1.5.5
@angular-devkit/build-optimizer: 0.0.34
@angular-devkit/core: 0.0.22
@angular-devkit/schematics: 0.0.38
@ngtools/json-schema: 1.1.0
@ngtools/webpack: 1.8.5
@schematics/angular: 0.1.8
typescript: 2.4.2
webpack-bundle-analyzer: 2.9.1
webpack: 3.8.1

$ ng build --prod
Time: 280909ms
chunk {0} polyfills.953b2ea9f096cadfdccc.bundle.js (polyfills) 146 kB [initial] [rendered]
chunk {1} main.b072d0d00d3efb0bb504.bundle.js (main) 1.78 MB [initial] [rendered]
chunk {2} styles.d1f7c9f576103c62cb95.bundle.css (styles) 108 kB [initial] [rendered]
chunk {3} inline.94db4a86d65f89eca9e2.bundle.js (inline) 1.49 kB [entry] [rendered]

$ ng build --prod --build-optimizer=false
Time: 178722ms
chunk {0} polyfills.50f5381850b537ae213f.bundle.js (polyfills) 147 kB [initial] [rendered]
chunk {1} main.c2e1bcb424c7762ae48a.bundle.js (main) 1.34 MB [initial] [rendered]
chunk {2} styles.d1f7c9f576103c62cb95.bundle.css (styles) 108 kB [initial] [rendered]
chunk {3} vendor.74317826779ffe91983b.bundle.js (vendor) 650 kB [initial] [rendered]
chunk {4} inline.3e5e45cce1262d7aea45.bundle.js (inline) 1.5 kB [entry] [rendered]

Only around 2 minutes difference now ！Thanks for your hard work @filipesilva.
If we can make the difference within 1 minute , so much the better :)
Thanks again.

[alexeagle]

From @filipesilva on December 1, 2017 11:49

All credit goes to @clydin, he's the one working on reducing Build Optimizer run time and memory usage.

[alexeagle]

From @caroline-church on December 14, 2017 16:17

Hi i am also having this issue but am not using angular-cli, any recomendations what i can do to make my build work with build-optimizer turned on. In my case it gets to around 50% then hangs forever

[alexeagle]

From @filipesilva on December 18, 2017 8:37

@caroline-church is your project open source or available for me to see somewhere?

[alexeagle]

From @caroline-church on December 18, 2017 9:16

@filipesilva here is my branch https://github.com/caroline-church/fabric-composer/tree/improve-performance-2. Its a mono-repo so you have to run npm install in the top level directory. The package you want to look at is composer-playground. If you run npm run buikd:prod you will see the problem. I narrowed it down to that the problem was the composer-common dependency, but wasn't sure why it would cause a problem.

[alexeagle]

From @pjmolina on December 22, 2017 16:57

Hi everyone!
I have a case when building for production an ng app:

This hangs forever (it is not a memory problem, uses around 400Mb) but keeps consuming CPU and do not finish.
ng build -e prod -prod -aot --base-href x

This other one, disabling the build optimizer, works:
ng build -e prod -prod -aot --base-href x --build-optimizer=false

One particularity on the ng project is that it contains one big/medium file (a generated parser) of 108 Kb in size and this looks to be the hanging point.

Versions used:

Angular CLI: 1.6.2
Node: 8.6.0
OS: win32 x64
Angular: 5.1.2
@angular/cli: 1.6.2
@angular-devkit/build-optimizer: 0.0.36
@angular-devkit/core: 0.0.22
@angular-devkit/schematics: 0.0.42
@ngtools/json-schema: 1.1.0
@ngtools/webpack: 1.9.2
@schematics/angular: 0.1.11
typescript: 2.5.3
webpack: 3.10.0

The project is not open source, cannot share it as is. But I can prepare a private repro sample if requested.

[alexeagle]

From @filipesilva on December 22, 2017 17:9

@pjmolina that's pretty interesting, the big file. Is the file itself something you can share?

[alexeagle]

From @pjmolina on December 22, 2017 17:11

@filipesilva Yes, in a private way (not for public disclosure).
I can sent it to you by e-mail.

[alexeagle]

From @pjmolina on December 22, 2017 19:8

@filipesilva I just created a shared full repro sample.

[alexeagle]

From @filipesilva on December 28, 2017 17:28

@pjmolina thanks a bunch for putting that repro up! I'll have a look to see if there's a specific build optimizer optimization that's causing problems.

/cc @clydin

[alexeagle]

From @filipesilva on December 28, 2017 18:0

I ran some tests disabling each individual optimization that Build Optimizer does and the culprit seems to be https://github.com/angular/devkit/blob/master/packages/angular_devkit/build_optimizer/src/transforms/prefix-classes.ts.

We're probably doing something horribly inefficient there and it takes a long time on big files.

[alexeagle]

From @pjmolina on December 28, 2017 18:14

Just in case, if the TypeScript compiler/lib is involved, as an hypothesis: Could it be related to this other bug: TypeScript #17033 I reported on the TS compiler?

[alexeagle]

From @filipesilva on December 28, 2017 18:29

Unsure... but something odd is happening here. I tried using the node inspector to get a CPU profile, and... it crashes.

I set a breakpoint right before Build Optimizer starts processing parser.ts and if I take a CPU profile then, for any amount of time (even something like 1s) and stop, the profiler will go up to 15gigs of ram trying to process it and crash chrome.

I might be related to that TS issue yes. Build Optimizer processes input files as JS files with allowJs. I double checked and it turns out the only applicable transform for the big file is prefix-classes so that's the one that ran. But now I tried forcing another transform for that file instead of prefix-classes and it also hangs.

So yeah, I think it boils down to microsoft/TypeScript#17033.

[alexeagle]

From @clydin on December 28, 2017 19:4

Did a quick test with PR #346 and a production build (ng build --prod) finishes in ~18 seconds. The culprit appears to be the type analysis phase of the TS compiler. The massive amount of nesting in the large parser file is probably the issue.

[alexeagle]

From @filipesilva on December 28, 2017 20:15

I think #346 is a great improvement, but worry about the sourcemap limitation. It means there are situations where having sourcemaps in production might mean extremely lengthy build. But that's the case now anyways.

[dhilgarth]

Is there any course of action to fix this issue?

[filipesilva]

@dhilgarth are you seeing some disproportionate build time increase? Overall Build Optimizer is expected to take make builds take longer, as it adds another step to the build pipeline.

In the past we have found bugs caused a disproportionate increase in build time however. To the best of my knowledge all those are currently fixed. But there might be unknown ones.

If you're in such a situation can you open a new issue with a reproduction please? This issue itself should be closed though, in the year since it was open we have fixed all outstanding Build Optimizer performance bugs.

[dhilgarth]

Thanks for the amazingly fast response.
Indeed, we are seeing build time increases by 200+% (14 minutes instead of 3), however, not consistently. And sometimes, it even seems to hang forever.

I am not sure if I can provide a repro as it is a rather big project. But, with the info that it should work, I will try to find the root cause and report it in a new issue.
Thanks so far

[filipesilva]

Can you show me your ng version output? I'd also be curious to know what the difference in bundle size is with and without Build Optimizer.

Although at this point if I had to bet on the root cause, it might be memory. If your builds are already close to the memory limit, anything that moves them closer will disproportionately increase build time due to GC churn. #5618 (comment) contains instructions on how to increase this limit.

[dhilgarth]

ng version on my developer machine:

Angular CLI: 7.2.3
Node: 8.9.4
OS: win32 x64
Angular: 7.2.3
... animations, cli, common, compiler, compiler-cli, core, forms
... http, platform-browser, platform-browser-dynamic
... platform-server, router

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.11.2
@angular-devkit/build-angular     0.11.2
@angular-devkit/build-optimizer   0.11.2
@angular-devkit/build-webpack     0.11.2
@angular-devkit/core              7.1.2
@angular-devkit/schematics        7.2.3
@angular/language-service         7.2.0
@ngtools/webpack                  7.1.2
@schematics/angular               7.2.3
@schematics/update                0.12.3
rxjs                              6.3.3
typescript                        3.1.6
webpack                           4.23.1

Build Agent, I currently have no way of getting that info, but it should be very similar as I am not aware that we installed angular-cli globally.

Thanks for the hint with memory. It might very well be the cause as our build agents are notoriously low on memory.

Regarding the difference in size:
Without optimizer:

chunk {0} runtime.ec2944dd8b20ec099bf3.js (runtime) 1.42 kB [entry] [rendered]
chunk {1} main.b3c0dacfbc701a03b9d5.js (main) 2.08 MB [initial] [rendered]
chunk {2} polyfills.6b272e87a923abcf68a5.js (polyfills) 155 kB [initial] [rendered]
chunk {3} styles.1fe3fedb6cd6276e06bc.css (styles) 572 kB [initial] [rendered]
chunk {4} vendor.dbd903576800650cc1a0.js (vendor) 6.56 MB [initial] [rendered]

With optimizer:

chunk {0} runtime.ec2944dd8b20ec099bf3.js (runtime) 1.41 kB [entry] [rendered]
chunk {1} main.4fd9b0921f00a8544bcb.js (main) 2.06 MB [initial] [rendered]
chunk {2} polyfills.6b272e87a923abcf68a5.js (polyfills) 154 kB [initial] [rendered]
chunk {3} styles.1fe3fedb6cd6276e06bc.css (styles) 572 kB [initial] [rendered]
chunk {4} vendor.cdfc37a510d0c613866d.js (vendor) 6.24 MB [initial] [rendered]

So a bit over 300kb

[dhilgarth]

BTW, we already encountered issues with the node memory limit, so we are using this command to run our build:
npm run ng-high-memory -- build --base-href=/XYZ/ --configuration=production

with ng-high-memory being defined as:
"ng-high-memory": "node --max_old_space_size=8000 ./node_modules/@angular/cli/bin/ng"

(So we actually already applied the hints from the linked issue)

[filipesilva]

I'd also update @angular-devkit/build-angular to the latest version btw. That's the actual build system. Memory is limited by NodeJS itself btw, so even on a machine with a lot of memory you won't be using it by default unless you use those scripts.

Also your total bundle might get quite smaller if you don't have the vendor bundle in production. Maybe try it if you're curious.

At the end of the day you do have over 8mb of code post optimizations... that's a lot of code. And it looks like the benefits from Build Optimizer might be marginal at for your code base. It is a reasonable option to turn it off if it's not worth the extra build time.

[dhilgarth]

Thanks for the hints. We actually did experiment without the vendor chunk and the differences in size were marginal, even more marginal than the difference with the build optimizer.

Regarding the overall size: You are correct, it's a lot. We are currently in the process of optimizing our dependencies, which hopefully shaves off some megabytes

[dhilgarth]

Now I do have such a hanging build again, after I updated @angular-devkit/build-angular. And the strange thing is that the node process uses about 1.4GB but no CPU at all. It just sits there, and idles. What could cause such behaviour? Is there some verbose logging that I could enable to troubleshoot this?

[filipesilva]

1.4gb is just at the memory limit. I'd say that's the reason.

[dhilgarth]

OK, but we are using ng-high-memory. Shouldn't this solve those kinds of problems?

[filipesilva]

It should indeed. I'm not sure what's happening in your project :/

I know some older versions of node had memory issues as well, so maybe node 10 helps but... being stuck at 1.4 makes it really look like the node memory flag isn't being applied. I've also seen it referenced as --max-old-space-size (dashes instead of underscore) but I'd be surprised if that was it.

[dhilgarth]

It's so strange. I just saw the memory go well over that limit, so the option is working. And now, the build worked again. But at times, it just doesn't.
Thanks for your help. If I will ever find out what's going on I will let you know.

[filipesilva]

Are you using Windows with WSL? I saw something similar once with WSL, where it really looked like the linux kernel just decided to kill off all the Terser child processes or something and it just hung there, doing nothing.

[dhilgarth]

Yes, I am currently on Windows 7, so there should be WSL there. And it matches what I saw:
When it worked, there was a child process that has been started and the main process had long stretches where it was idle.
When it didn't work, there was no child process, so it could very well be that it was killed. I will keep an eye on that.

[angular-automatic-lock-bot]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}