Memory usage regression related to uglify between Webpack 2.1 beta to 3.X

[webpack-bot]

Do you want to request a feature or report a bug?

Report a bug

What is the current behavior?

Using the uglify plugin on a large project (3000-4500 modules) to do a production build seems to have fairly high memory usage and fails with an an heap out of memory error. It's not 100%, but in our projects is fairly easy to repro in all versions I tried, 2.7 all the way to the latest 3.X. In the older 2.1 (beta) version, it never happened over hundreds of builds.

I've tried to narrow down the exact change that causes this but have been unable to do so.

If the current behavior is a bug, please provide the steps to reproduce.

Using a fairly large build (3000+ modules), we're using the following config for the Uglify plugin. Removing it stops the issue from happening as far as I can tell. Sourcemaps enabled are not necessary to repro but make it more frequent:

      new webpack.optimize.UglifyJsPlugin({
        compress: {
          warnings: false,
          reduce_vars: false,
          sequences: false,
        },
        output: {
          comments: false,
        },
        sourceMap: true,
      })

What is the expected behavior?

So just using --max_old_space_size=XXXX fixes the issue and normally I'd just punt on this saying "our build is big, so it's normal", but the fact that it never happened in 2.1 leads me to think there's a regression somewhere.

If this is a feature request, what is motivation or use case for changing the behavior?

Please mention other relevant information such as the browser version, Node.js version, webpack version and Operating System.

Happens in both node 6 LTS and Node 8 LTS, Webpack 2.7 as well as 3.X, MacOS as well as Linux, across several applications (but is related to their size)

---

This issue was moved from webpack/webpack#6048 by @evilebottnawi. Orginal issue was by @Phoenixmatrix.

[alexander-akait]

Need review latest version on memory leak

[alexander-akait]

@Phoenixmatrix problem still exists?

[Phoenixmatrix]

As far as I know it exists on everything at least 2.7 and up all the way to latest(I have not tested all versions between the 2.X-beta and 2.7)

I have not tested 4.X alpha as I don't have any app large enough to trigger the issue that will work in 4.X

[michael-ciniawsky]

@Phoenixmatrix

1. Are you using the ModuleConcatenationPlugin() in production, or what other plugins are included exclusively for a production build ?
2. Does this also happen without using uglifyjs-webpack-plugin (in general).
3. Could you try with uglifyjs-webpack-plugin >= v1.0.0and see if it still happens

[Phoenixmatrix]

1. While we do use a very different configuration in production, when I did my tests that narrowed things down to Ugilify, I was not using anything else from our production setup (I was just toggling UglifyJSPlugin). We don't use ModuleConcatenationPlugin right now.

2. It doesn't. For what its worth, the heap out of memory error when this happens always happens in a way that shows a a few stack frames from within the Uglify Plugins' code (the exact stack frames are pretty much random, but it always happens there).

3. I did not try that yet, only the version that Webpack itself depends on (0.4.6) , so I'll give v1 a shot.

[michael-ciniawsky]

While we do use a very different configuration in production, when I did my tests that narrowed things down to Ugilify, I was not using anything else from our production setup (I was just toggling UglifyJSPlugin). We don't use ModuleConcatenationPlugin right now.

It doesn't. For what its worth, the heap out of memory error when this happens always happens in a way that shows a a few stack frames from within the Uglify Plugins' code (the exact stack frames are pretty much random, but it always happens there).

👍

I did not try that yet, only the version that Webpack itself depends on, so I'll give v1 a shot.

Yep, please report back if this works (better) and include the stack trace / V8 OOM Error in case it still happens

[kzc]

Using a fairly large build (3000+ modules),

This is not surprising with such a large code base. Source maps in particular chew up a ton of memory.

Your build was likely very close to the node memory limit before and some new code optimizations in more recents versions of webpack and uglify likely pushed it over the limit.

It looks like webpack v2.1.0-beta.28 used uglify-js@2.7.x

webpack@3.10.0 uses uglifyjs-webpack-plugin@0.4.6 which uses uglify-js@2.8.29

If you want to isolate uglify memory use just remove the uglify plugin statements from your webpack configs for the two webpack versions and then post the complete unminified webpack produced bundles in a couple of gists so they can be examined.

To get faster uglify builds using less memory see:

https://github.com/mishoo/UglifyJS2#uglify-fast-minify-mode

and disable source maps.

[Phoenixmatrix]

This is not surprising with such a large code base. Source maps in particular chew up a ton of memory.

Yeah, if it didn't go from happening 0% of the time across tens of thousands of builds, to happening 2/3rd time across a dozen different apps (we have hundreds, but only the big ones experienced it, though with consistency), I wouldn't have reported anything.

[kzc]

@Phoenixmatrix It would be helpful if you could provide peak webpack memory usage and wall clock timings for the following scenarios similar to this issue using Task Manager for Windows or top for UNIX:

1. webpack@2.1, uglify enabled, source maps enabled
2. webpack@2.1, uglify disabled, source maps enabled
3. webpack@2.1, uglify enabled, source maps disabled
4. webpack@2.1, uglify disabled, source maps disabled
5. webpack@3.10, uglify enabled, source maps enabled
6. webpack@3.10, uglify disabled, source maps enabled
7. webpack@3.10, uglify enabled, source maps disabled
8. webpack@3.10, uglify disabled, source maps disabled

Use a sufficiently high --max_old_space_size=XXXX value so these processes do not run out of memory. Build the same source code in all scenarios.

[andreialecu]

This might be related: mishoo/UglifyJS#2609

[kzc]

@andreialecu In mishoo/UglifyJS#2609 you wrote:

Regarding the memory leak, I was mistaken, there seems to be none when running webpack normally. My apologies.

This is a different issue.

[Phoenixmatrix]

Update: I didn't get to run tests yet but will do so this week (holidays are a little crazy). I'm not forgetting about this.

[kzc]

  new webpack.optimize.UglifyJsPlugin({

This shows that uglify-js@2 was used in this issue, not uglify-es. So it is not related to mishoo/UglifyJS#2609.

[Phoenixmatrix]

Ok, that took forever (building our apps takes a bit, and making sure everything lined up properly with Lerna for all the tests was a pain, especially for testing the older webpack version, but I got it right in the end).

Methodology

I took one of our above average size apps that had experienced the issue (not the largest ones, because it was taking too long to build to be practical). The only difference between the tests are:

• Version of webpack (3.10.0 vs 2.1 beta)
• Version of the uglify plugin (only with webpack 3.10.0. I used 1.1.4 vs whatever is bundled with Webpack...0.4.X i think)
• I included the uglify plugin directly (I did not use webpack optimize-minimize switch to ensure I controlled what changed)
• The only differences across run were if sourcemaps were enabled or not (the uglify plugin option, NOT the webpack devtool or loader options. Those stayed the same), as well as if the uglify plugin was included at all or not. Nothing else changed (I used our production build setup in every case for all other settings). Note that this means I did not test uglify disabled and sourcemap enabled, because it would not make sense with this methodology
• This was all on Mac OS El Captain (don't ask why I haven't upgraded yet)
• To gather the peak memory usage, I used usr/bin/time -l and recorded the maximum resident set size. This isn't ideal (the problem is with heap memory usage) as it's very dependent on garbage collection behavior, but it was easy and quick to do.
• All runs were done with the --max-old-space-size=4096 switch for node (on our projects if I use the default the builds randomly crash on failed heap allocations. Non-deterministic).
• Because of time constrains, I did NOT do a lot of runs with each setup. I ran everything at least twice and the results were within a few megabytes of each other though. Not very scientific, I know :)

Results

Webpack version	Uglify enabled	Sourcemaps enabled	Peak memory
3.10.0 w uglify plugin 1.1.4	Yes	Yes	3090mb
3.10.0 w uglify plugin 1.1.4	Yes	No	1710mb
3.10.0 w uglify plugin 1.1.4	No	No	1228mb
3.10.0	Yes	Yes	3224mb
3.10.0	Yes	No	1917mb
3.10.0	No	No	1323mb
2.1	Yes	Yes	3067mb
2.1	Yes	No	2151mb
2.1	No	No	1400mb

Conclusion

The results are pretty close and within the range of fluctuation that garbage collection can create, so it's not very enlightening. With that said, the uglify version that webpack 3.10 depends on seems to use a bit more memory with uglify + sourcemaps than the older version or the newer version. Enough to explain the random heap allocation errors we would see.

Next I'll try putting 1.1.4 in our real production setup and see what happens.

[kzc]

Thanks for the data. I don't see any problem here. Uglifying code in any configuration for your sources always takes an additional 500 to 600mb of RAM. It's well known that source maps consume a lot of memory.

[Phoenixmatrix]

Of course. Again, the issue is that moving from webpack 2.1 to 2.7/3.X we saw our memory usage go up enough that we went from 0% failure rate to 50%~ failure rate over heap allocation related crashes in our bigger projects, with everything else (including sourcemaps) being the same.

[kzc]

As I thought with my comment above:

Your build was likely very close to the node memory limit before and some new code optimizations in more recents versions of webpack and uglify likely pushed it over the limit.

[kzc]

I think this issue can be closed as there's no significant memory regression, and the chart above shows that memory used by uglifyjs-webpack-plugin@1.x is actually lower than uglifyjs-webpack-plugin@0.4.6.