Understand why Webpack is running out of Memory

[daviddias]

Currently, CI fails with:

Test Browser
<--- Last few GCs --->
  332336 ms: Mark-sweep 1341.7 (1418.5) -> 1335.0 (1416.2) MB, 403.5 / 0.0 ms [allocation failure] [GC in old space requested].
  332750 ms: Mark-sweep 1335.0 (1416.2) -> 1335.0 (1416.2) MB, 414.8 / 0.0 ms [allocation failure] [GC in old space requested].
  333142 ms: Mark-sweep 1335.0 (1416.2) -> 1335.0 (1401.2) MB, 391.3 / 0.0 ms [last resort gc].
  333522 ms: Mark-sweep 1335.0 (1401.2) -> 1335.0 (1401.2) MB, 380.5 / 0.0 ms [last resort gc].
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x25fbd6fcf781 <JS Object>
    1: toString [buffer.js:490] [pc=0x83681c5517c] (this=0x3368e86806a9 <an Uint8Array with map 0x580e41ffe69>)
    2: /* anonymous */(aka /* anonymous */) [/home/travis/build/ipfs/js-ipfs-api/node_modules/karma-webpack/lib/karma-webpack.js:255] [pc=0x83681e0f89f] (this=0x25fbd6f04381 <undefined>,err=0x25fbd6f04381 <undefined>,content=0x3368e86806a9 <an Uint8Array with map 0x580e41ffe69>)
    3...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [node]
 2: 0x10a03cc [node]
 3: v8::Utils::ReportApiFailure(char const*, char const*) [node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
 5: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [node]
 6: v8::internal::Factory::NewStringFromUtf8(v8::internal::Vector<char const>, v8::internal::PretenureFlag) [node]
 7: v8::String::NewFromUtf8(v8::Isolate*, char const*, v8::String::NewStringType, int) [node]
 8: node::StringBytes::Encode(v8::Isolate*, char const*, unsigned long, node::encoding) [node]
 9: void node::Buffer::StringSlice<(node::encoding)1>(v8::FunctionCallbackInfo<v8::Value> const&) [node]
10: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) [node]
11: 0x9ee7fe [node]
12: 0x9ef09e [node]
13: 0x8367fb092a7
Aborted (core dumped)

This issue was created with the upgrade to the new ipfsd-ctl. The cause is that WebPack is trying to bundle multiple versions of the ipfs-api into the client that spawns nodes in the browser, making it run out of memory.

@dryajov wanna take the lead on fixing this given your work on ipfsd-ctl?

[dryajov]

sure thing, taking a look now.

[daviddias]

@dryajov any progress? Not having CI fully green is troublesome.

[daviddias]

@dryajov I just noticed none of the tests is using the ipfsd-ctl server to spawn the nodes. All the tests are trying to spawn a daemon from their own process.

See #688

[dryajov]

@diasdavid by default, when running in node it will spawn a process directly, when running in browser it will spawn it using the server. It should work as expected right now. I don't believe we need #688. Take a look at the remote parameter here - https://github.com/ipfs/js-ipfsd-ctl#daemon-factory-class

[daviddias]

Temporary fix with #688

[daviddias]

@victorbjelkholm just saw a OOM in Jenkins too:

url https://ci.ipfs.team/blue/organizations/jenkins/IPFS%2Fjs-ipfs-api/detail/PR-700/4/pipeline

Apparently it just happens some of the times.

[victorb]

@diasdavid was this after https://github.com/ipfs/js-ipfs-api/pull/688/files was merged? In that case, our --max-old-space-size=8192 fix doesn't actually fix the issue...

Also, anectodotal, I've been building some pretty large applications with webpack, and never hit this issue before. Are we sure we are not doing something silly when building?

[victorb]

Tried building the module itself with aegir, and it works even if setting max-old-space-size to 512 so definitly something weird with the tests.

[victorb]

Disabling source maps gives a bit better insight into a possible cause, a very long string!

[vmx]

I hope that this is a viable solution to the problem: ipfs/aegir#203

[daviddias]

@vmx solved this <3 Thank you!!