Plugin crashes when trying to stringify a large stat blob

[evan-scott-zocdoc]

Not sure how big it starts to fail to be honest, but I don't think it's a massive number in terms of MB. It might make sense to adjust this plugin to use a streaming JSON stringifier and async write to disk to help overcome this?

Manifests itself commonly when using ModuleConcatenation on a webpack project with multiple entry points.

[valscion]

What kind of a crash are you seeing? Would it be possible for you to generate an example repository that is big enough to demonstrate this crash? It would help us in making sure that we've fixed the problem.

[evan-scott-zocdoc]

It’s Node running out of memory when calling JSON.stringify due to the size of the stats object. Closed source and a big project, so I can’t share it unfortunately.

…

________________________________ From: Vesa Laakso <notifications@github.com> Sent: Sunday, October 1, 2017 6:26:23 AM To: th0r/webpack-bundle-analyzer Cc: Evan Scott; Author Subject: Re: [th0r/webpack-bundle-analyzer] Plugin crashes when trying to stringify a large stat blob (#119) What kind of a crash are you seeing? Would it be possible for you to generate an example repository that is big enough to demonstrate this crash? It would help us in making sure that we've fixed the problem. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#119 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AYZrxpJ7basFIK6t_McY73BrWPyhbxD6ks5sn2jPgaJpZM4PpAUS>.

[valscion]

Closed source and a big project, so I can’t share it unfortunately.

That's why I said generate in "Would it be possible for you to generate an example repository". It would improve the possibility of getting this fixed in a reasonable time frame.

[hlehmann]

I have the same since recently, when I added ModuleConcatenationPlugin. I get a Very long string error.

Here the stack.

<--- Last few GCs --->

[26459:0x375e5e0]    80551 ms: Mark-sweep 1075.2 (1414.3) -> 1074.9 (1401.8) MB, 169.9 / 0.0 ms  allocation failure GC in old space requested
[26459:0x375e5e0]    80744 ms: Mark-sweep 1074.9 (1401.8) -> 1074.5 (1396.3) MB, 192.8 / 0.0 ms  allocation failure GC in old space requested
[26459:0x375e5e0]    80949 ms: Mark-sweep 1074.5 (1396.3) -> 1074.4 (1328.8) MB, 204.6 / 0.0 ms  last resort 
[26459:0x375e5e0]    81153 ms: Mark-sweep 1074.4 (1328.8) -> 1074.4 (1299.3) MB, 204.7 / 0.0 ms  last resort 


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2271f1728799 <JSObject>
    2: fromString(aka fromString) [buffer.js:308] [bytecode=0x14e14f1cd6a1 offset=202](this=0x3f71f6082311 <undefined>,string=0xe91e2c0a269 <Very long string[195551808]>,encoding=0x2271f1738331 <String[4]: utf8>)
    4: from [buffer.js:173] [bytecode=0x14e14f1cd181 offset=11](this=0x11110d931559 <JSFunction Buffer (sfi = 0x3c6853a13ab1)>,value=0xe91e2c0a269 <Very long string[195551808]>,encoding...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [node]
 2: 0x1356bec [node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
 5: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [node]
 6: v8::internal::String::SlowFlatten(v8::internal::Handle<v8::internal::ConsString>, v8::internal::PretenureFlag) [node]
 7: v8::internal::String::Flatten(v8::internal::Handle<v8::internal::String>, v8::internal::PretenureFlag) [node]
 8: v8::String::WriteUtf8(char*, int, int*, int) const [node]
 9: node::StringBytes::Write(v8::Isolate*, char*, unsigned long, v8::Local<v8::Value>, node::encoding, int*) [node]
10: node::Buffer::New(v8::Isolate*, v8::Local<v8::String>, node::encoding) [node]
11: 0x137b414 [node]
12: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) [node]
13: 0xc13f5a [node]
14: v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) [node]
15: 0x20572c1846fd

[hlehmann]

No issue when using generateStatsFile: false

[valscion]

Ohh, so this issue is only happening when you generate a stats.json file? And things work a-OK if generateStatsFile: false, right?

[hlehmann]

absolutly it crashes with generateStatsFile: true but not with generateStatsFile: false. I have no issue to get the html report.

current config:

new BundleAnalyzerPlugin({
  analyzerMode: 'static',
  reportFilename: __dirname + '/../build-report.html',
  generateStatsFile: false,
  statsFilename: __dirname + '/../build-stats.json',
}),

[valscion]

Ok, that's relieving to hear! I wasn't sure what part of the plugin crashed, but if it was just the stats stringifying, then this should be pretty straightforward to fix.

We're open to accept a PR fixing this. The code in question is this:

webpack-bundle-analyzer/src/BundleAnalyzerPlugin.js

Lines 64 to 72 in e6fbbe7

	generateStatsFile(stats) {
	const statsFilepath = path.resolve(this.compiler.outputPath, this.opts.statsFilename);
	mkdir.sync(path.dirname(statsFilepath));
	fs.writeFileSync(
	statsFilepath,
	JSON.stringify(stats, null, 2)
	);

It should be OK to mark that function as async and await the completion of JSON stringify stream. Let me know if you need any help figuring things out and I'll help to the best of my knowledge!

[valscion]

@evan-scott-zocdoc and @hlehmann: Could you test PR #129 by @ryan953 on your local project to see if it fixes your issues?

wget http://vesalaakso.com/webpack-bundle-analyzer-2.9.0-pr129.tgz
npm install webpack-bundle-analyzer-2.9.0-pr129.tgz

...and then try to generate the stats file?

[evan-scott-zocdoc]

@valscion just tried it out in our biggun project and it worked! Awesome job @ryan953, thanks!

[valscion]

Fix by @ryan953 has been released in v2.10.0 🎉. Upgrading might be a good idea 😉