Webpack build hangs with parallel: true

[Pajn]

• Operating System: Windows 10 1810, WSL, Ubuntu 18.04
• Node Version: 8.12.0
• NPM Version: 6.4.1
• webpack Version: 4.19.1
• terser-webpack-plugin Version: 1.1.0

Expected Behavior

That the webpack build completes

Actual Behavior

The webpack build hangs forever (or at least 30 minutes which is the longest I have lasted) after outputting

Compilation  starting…

However changing parallel: true to parallel: false in terser options will cause the build to complete in normal time (around 40s)

Code

new TerserPlugin({
        terserOptions: {
          parse: {
            // we want terser to parse ecma 8 code. However, we don't want it
            // to apply any minfication steps that turns valid ecma 5 code
            // into invalid ecma 5 code. This is why the 'compress' and 'output'
            // sections only apply transformations that are ecma 5 safe
            // https://github.com/facebook/create-react-app/pull/4234
            ecma: 8
          },
          compress: {
            ecma: 5,
            warnings: false,
            // Disabled because of an issue with Uglify breaking seemingly valid code:
            // https://github.com/facebook/create-react-app/issues/2376
            // Pending further investigation:
            // https://github.com/mishoo/UglifyJS2/issues/2011
            comparisons: false,
            // Disabled because of an issue with Terser breaking valid code:
            // https://github.com/facebook/create-react-app/issues/5250
            // Pending futher investigation:
            // https://github.com/terser-js/terser/issues/120
            inline: 2
          },
          mangle: {
            safari10: true
          },
          output: {
            ecma: 5,
            comments: false,
            // Turned on because emoji and regex is not minified properly using default
            // https://github.com/facebook/create-react-app/issues/2488
            ascii_only: true
          }
        },
        // Use multi-process parallel running to improve the build speed
        // Default number of concurrent runs: os.cpus().length - 1
        parallel: true,
        // Enable file caching
        cache: true,
        sourceMap: shouldUseSourceMap
      })

Full webpack config: https://github.com/Pajn/tscomp/blob/cra2/config/webpack.config.prod.js

How Do We Reproduce?

I suspect this can be related to WSL. If you have some debugging tips I can probably run them.

For the exact same state as I have:

1. Clone https://github.com/Pajn/tscomp
2. git checkout terser-bug
3. Clone https://github.com/Pajn/RAXA
4. cd RAXA/packages/web
5. yarn
6. yarn add --dev path/to/cloned/tscomp/repo
7. yarn build, or for more information NODE_ENV=production webpack --config node_modules/tscomp/config/webpack.config.prod.js --verbose --info-verbosity verbose --display verbose --profile

[alexander-akait]

@Pajn Thanks for issue, we have some problem on WSL for uglify plugin webpack-contrib/uglifyjs-webpack-plugin#302, seems problem in worker-farm, thanks for reproducible test repo, i am not familiar with WSL, can you provide instruction how i can setup this env?

[Pajn]

To set up WSL you need Windows 10, then you should be able to follow this guide https://docs.microsoft.com/en-us/windows/wsl/install-win10

After that you can basically treat is as a normal Ubuntu Linux so install node by following this https://nodejs.org/en/download/package-manager/#debian-and-ubuntu-based-linux-distributions
Git can be installed via sudo apt install git

The only consideration with a case like this is that Linux files are not visible to Windows, so to be able to open files in a graphical editor you need to save them under /mnt/c

[alexander-akait]

@Pajn oh, i don't have windows 10 right now, can debug this only on next week, feel free to debug problem self. Here default arguments to work-farm https://github.com/webpack-contrib/terser-webpack-plugin/blob/master/src/TaskRunner.js#L36, maybe we need improve this check process.platform === 'win32' and pass maxConcurrentCallsPerWorker: 1 on WSL

[Pajn]

You are right that worker-farm is somehow related.
This is what I get from pstree in a stuck build

  │   │   ├─node /usr/share/yarn/bin/yarn.js build
  │   │   │   ├─sh -c tscomp build
  │   │   │   │   └─node /c/Users/Rasmus/Development/RAXA/packages/web/node_modules/.bin/tscomp build
  │   │   │   │       ├─node /c/Users/Rasmus/Development/RAXA/packages/web/node_modules/tscomp/scripts/build.js
  │   │   │   │       │   ├─node /c/Users/Rasmus/Development/RAXA/packages/web/node_modules/tscomp/node_modules/worker-farm/lib/child/index.js /usr/bin/node /c/Users/Rasmus/Development/RAXA/packages/web/node_modules/tscomp/scripts/build.js
  │   │   │   │       │   │   └─5*[{node}]
  │   │   │   │       │   ├─node /c/Users/Rasmus/Development/RAXA/packages/web/node_modules/tscomp/node_modules/worker-farm/lib/child/index.js /usr/bin/node /c/Users/Rasmus/Development/RAXA/packages/web/node_modules/tscomp/scripts/build.js
  │   │   │   │       │   │   └─5*[{node}]
  │   │   │   │       │   └─9*[{node}]
  │   │   │   │       └─5*[{node}]
  │   │   │   └─9*[{node}]

Changing that condition so that maxConcurrentCallsPerWorker: 1 is passed (process.platform is linux) did not help unfortunately.

I tried running the basic example from the worker-farm readme and that worked without hanging so I guess there some problem with file locking.

I'll continue the debugging tomorrow.

[alexander-akait]

@Pajn thanks for helping! How many cpu you are have? Based on output worker-farm is create child processes but don't get response.

[Pajn]

I have four, however the above pstree output is from a run where I limited the number of workers to two, just to see if it did help.

I modified the code so that it would still use worker-farm with only one worker, and then it did finish again. I also tried to attach strace to see if I got some hint of the problem but did not find anything useful in the output. If you are interested I can of course share those outputs.

[alexander-akait]

@Pajn Feel free to publish outputs

[alexander-akait]

Guys i try to reproducible problem but i can't, maybe can create really minimum reproducible test repo with really minimum setup?

[charsleysa]

@evilebottnawi you should be able to recreate this by compiling the an Angular demo app (any should do) with aot and optimization flags turned on.

I get this issue where the TerserPlugin seems to hang with a lot of node processes showing in Task Manager using 0% CPU resource.

[alexander-akait]

@charsleysa can you provide instruction for minimum reproducible test repo?

[charsleysa]

@evilebottnawi it seems to be something to do with having a project sufficiently large enough producing source maps on a production build.

The following instructions assume you have already setup WSL with Node 10.x (node inside WSL) using standard setup instructions. The below instructions are executed inside bash.

1. clone this repo https://github.com/formio/angular-demo
2. cd into the cloned repo directory
3. install the angular cli npm install -g @angular/cli
4. install the repo requirements npm install
5. run a standard build just to check that everything is working ng build
6. run a production build with source maps turned on ng build --configuration=production --project=demo --source-map=true
7. if you watch Task Manager you'll notice that eventually the node processes will just cease utilizing CPU resource with the build reporting 92% chunk asset optimization TerserPlugin (as pictured below)

[alexander-akait]

@charsleysa thanks for instructions, anyway feel free to investigate problem(s), i think problem inside worker-farm, maybe you can find problem faster than i can, because i need time to setup WSL

[andreialecu]

WSL is becoming more and more popular. I'm seeing the same problem as well and can confirm that parallel: false works. If #6 is applied to make parallel the default, then this needs to be fixed first.

[alexander-akait]

@andreialecu can you investigate why it is happens? Don't have WSL around and don't have windows machine near, maybe you can find problem

[andreialecu]

I've been digging at it and from my initial findings it seems like this callback is never ran:

https://github.com/webpack-contrib/terser-webpack-plugin/blob/master/src/TaskRunner.js#L74

However, the task itself and the code in minify.js is called, and is returning properly. I'm trying to investigate why it never gets there now. I suspected it would be a problem in worker-farm but the examples in the worker-farm repo and all of its tests pass properly.

[alexander-akait]

@andreialecu yep, problem in worker-farm, maybe we need to setup more options (https://github.com/webpack-contrib/terser-webpack-plugin/blob/master/src/TaskRunner.js#L34) for WSL

[alexander-akait]

What is process.platform on WSL?

[andreialecu]

But like I said, worker-farm's entire test suite passes properly.

[andreialecu]

Further progress on debugging this, it seems like the forked process' on('message') handler in worker-farm is never called when running with this plugin on WSL:

https://github.com/rvagg/node-worker-farm/blob/master/lib/farm.js#L145

However, when running worker-farm outside of this scenario, it is called. I verified using this: https://github.com/rvagg/node-worker-farm/tree/master/examples/basic

What could be different?

[alexander-akait]

@andreialecu only setup this options https://github.com/webpack-contrib/terser-webpack-plugin/blob/master/src/TaskRunner.js#L34 (as i written above)

[andreialecu]

I think I found the root cause.

It is related to the length of the result being passed from the worker process back to the farm via process.send(). If I make it send a simple 'hello' instead of the entire minified script result it seems that all callbacks are called.

Making this callback with a shorter string results in all callbacks working:
https://github.com/webpack-contrib/terser-webpack-plugin/blob/master/src/worker.js#L17

I'm trying to reproduce it here https://github.com/rvagg/node-worker-farm/blob/master/examples/basic/index.js by changing the code to send a 10MB result and it works perfectly:

module.exports = function (inp, callback) {
  const msg = 'a'.repeat(10 * 1000 * 1000);
  callback(null, inp + ' BAR (' + process.pid + ')' + msg)
}

So I'm not sure what is going on.

[alexander-akait]

@Pajn Maybe some characters break this 😕

[andreialecu]

@evilebottnawi Well, I just tested that by dumping the objects returned by the minify() worker functions to local .json files then trying to read each of them in simple workers in the basic example above, and then pass them on to the main process. Everything worked.

I'm stumped.

Perhaps someone can take over. Running out of time assigned to this right now.

FWIW: as a quick test, I tried calling back with the first 100000 characters of JSON.stringify(minify(...)) from the minify worker and those went through, but the first 1000000 characters made it hang.

[alexander-akait]

weird 😕

[andreialecu]

A possible workaround for this would be to save all minified scripts to temp files from the worker and call back with the path to the temp files, then load them back in the master. This would probably fix it. How would you feel about this?

[alexander-akait]

@andreialecu as workaround sounds good, but will be great find real problem why it is doesn't work as expected, maybe bug in WSL and we should report about this bug. Because on macos/linux/windows without WSL it is works as expected. So i think problem in WSL. If problem won't fix we implement workaround. Write files in temp directory decrease perf and it is not good.

[joewood]

When this works through a workaround, is the temp directory in /tmp or is it under /mnt/c/ ? I'm wondering if this is a file system related bug.

[alexander-akait]

@andreialecu reproduce problem ⭐ WIP on fix

[endiliey]

I am on 1803. Surprisingly it worked now for my repo now on WSL😃.Previously it wasn't working, had to manually disable parallel

Edit: let me try the repro test
Edit 2: Doesn't work on the #21 (comment)

Very interesting because it is working on my local repo that once had this problem, but still not working on the reproducible repo. Could it be a dependency problem ?

[alexander-akait]

Repo #21 (comment) is very hard 😞 i was spent around hour to search configuration and no luck, in future all big repositories will be ignored, it is hard to debug and it spends a lot of time

[endiliey]

I think #21 (comment) webpack config is defined/ constructed dynamically on @angular/cli which is another huge project.

[alexander-akait]

@endiliey @andreialecu looks it is not freeze, try to wait 5-25 minutes (depends on you hardware), problem what source map generation takes a lot of time

[alexander-akait]

In webpack@5 we update https://github.com/mozilla/source-map to latest version (WASM) and it should solve problem with perf, but here we can't do this 😞

[alexander-akait]

Tried on many repos and all works fine, in some repos i have waited around 20 minutes due it is really big application. Just ensure it is not problem in plugin with WSL, just disable source map and try again

[andreialecu]

Did you see 0% cpu for node processes for a very long time while you waited, and it eventually finished? Did you try with parallel: false?

A problem still exists on WSL that is not present for native Windows or native Linux builds. If you go through my comments above you can use a debugger to attach and break on something inside the terser plugin code even if it is running via ng build.

Anyway, it's probably not worth working too much on this because I assume that it will automagically fix itself on WSL 2.0, and in the mean time parallel: false works without huge performance penalties.

[endiliey]

I have been running it for 2 hours but it still stucks in 92%. Also 0% node process

[alexander-akait]

@endiliey @andreialecu it is very strange, 20 min for #21 (comment) and all were minified, maybe you can create other minimum reproducible test repo?

[alexander-akait]

I am on SSD, maybe WSL has timeouts and in some cases it is failed 😕

[alexander-akait]

I can disable parallels on WSL by default, but i want to ensure it is real problem in WSL, not in webpack or other ecosystem

[alexander-akait]

Okey, we disable parallels on WSL and for 2.0 we migrate on jest-worker (they work with WSL) #89

[alexander-akait]

Somebody can test master version on WSL?

[alexander-akait]

Can't setup azure pipelines for WSL, if anyone had experience please put here 😄

[alexander-akait]

/cc @Pajn can you test terser-webpack-plugin@2 on WSL? Thanks

[Pajn]

I'm sorry. I gave up on trying to use Windows for development because problems like this was the norm and not an exception so now I only have Linux to test on.

[alexander-akait]

/cc @andreialecu friendly ping, you are still on WSL?

[alexander-akait]

I am closing issue, if somebody will get an error, just open a new problem with a minimal example, thanks

[SebastienGllmt]

I spent like an hour trying to figure out why Terser refused to run in parallel no matter what I did only to realize that parallel is disabled if you are on WSL. I'm currently on Windows build 19041 (Slow ring insider preview) using WSL2 and it works perfectly with all cores.

[jerrodrs]

Upgraded my terser plugin from 1.3.5 to 2.3.5 and the issue is resolved, can use parallel now.