Gatsby@2.20.29 Build failure (JavaScript heap out of memory)

[vai0]

Description

When running gatsby build, the build fails with the following error:

⠴ Building production JavaScript and CSS bundles
[                            ]   0.729 s 1/447 0% run queries

<--- Last few GCs --->

[13967:0x3e84850]   213180 ms: Scavenge 1350.6 (1417.9) -> 1350.1 (1418.4) MB, 2.6 / 0.0 ms  (average mu = 0.235, current mu = 0.188) allocation failure 
[13967:0x3e84850]   213184 ms: Scavenge 1351.0 (1418.4) -> 1350.5 (1419.4) MB, 2.2 / 0.0 ms  (average mu = 0.235, current mu = 0.188) allocation failure 
[13967:0x3e84850]   213187 ms: Scavenge 1351.2 (1419.4) -> 1350.9 (1419.9) MB, 2.3 / 0.0 ms  (average mu = 0.235, current mu = 0.188) allocation failure 


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x1552c375be1d]
    1: StubFrame [pc: 0x1552c37134b0]
Security context: 0x0f543369e6e9 <JSObject>
    2: /* anonymous */(aka /* anonymous */) [0x388649060421] [/home/jchi/projects/bifrost/frontend/node_modules/webpack-sources/lib/OriginalSource.js:~43] [pc=0x1552c726a4f9](this=0x147da5d026f1 <undefined>,item=0x388649060391 <String[17]\:       case "up":\n>)
    3: arguments adaptor frame: 3->1
    4: map [0xf5433686e...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x8e8c18 node::Abort() [node]
 2: 0x8e8c61  [node]
 3: 0xac0a02 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xac0d58 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xe97ad2  [node]
 6: 0xea78c3 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 7: 0xea81c6 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 8: 0xeaa911 v8::internal::Heap::AllocateRawWithLigthRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
 9: 0xeaa972 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
10: 0xe75594 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [node]
11: 0x110a3f5 v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [node]
12: 0x1552c375be1d 
Aborted
make[1]: *** [Makefile:125: npm-build] Error 134
make[1]: Leaving directory '/home/jchi/projects/bifrost'
make: *** [Makefile:33: build-frontend] Error 2

This issue began occurring after updating gatsby from 2.18.17 to 2.20.29.

Steps to reproduce

Run gatsby build --open-tracing-config-file node_modules/gatsby/dist/utils/tracer/jaeger-local.js

Unfortunately I am unable to share the repo, but I can share the jaegar build trace generated from the instructions here.

Here's a screenshot of the last 3 rows in the trace before it crashes:

If you notice the second to last 2 are hung up on the onCreateWebpackConfig hook in gatsby-plugin-netlify-cms.

At first I thought it was because my gatsby-plugin-netlify-cms was not properly updated because the issue seems to have been resolved, but it's now resurfaced after upgrading gatsby.

Here's the relevant package versions my project is running:

// package.json
...
"gatsby": "2.20.29",
"gatsby-plugin-netlify-cms": "4.2.4",
"netlify-cms-app": "2.12.11",
...

and my gatsby-config.js with some custom webpack configurations:

// gatsby-config.js
...
{
    resolve: "gatsby-plugin-netlify-cms",
    options: {
        ...
        customizeWebpackConfig: (config, { plugins }) => {
            config.resolve = {
                mainFields: ["browser", "main", "module"],
                modules: [
                    path.resolve(__dirname, "src"),
                    "node_modules",
                ],
            };

            config.plugins.push(
                plugins.define({
                    PRERENDER_NAVBAR: JSON.stringify(false),
                    "process.env": {
                        NODE_ENV: JSON.stringify(process.env.NODE_ENV),
                    },
                })
            );

            config.devtool = false;
        },
    },
},
...

Expected result

The expected result is the build should successfully complete.

Actual result

The build fails before completion.

Environment

Run gatsby info --clipboard in your project directory and paste the output here.

gatsby info --clipboard
                                                                                                                               
  System:
    OS: Linux 4.19 Debian GNU/Linux 10 (buster) 10 (buster)
    CPU: (8) x64 Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
    Shell: 5.7.1 - /usr/bin/zsh
  Binaries:
    Node: 10.15.3 - ~/.nvm/versions/node/v10.15.3/bin/node
    Yarn: 1.21.1 - ~/.nvm/versions/node/v10.15.3/bin/yarn
    npm: 6.4.1 - ~/.nvm/versions/node/v10.15.3/bin/npm
  Languages:
    Python: 3.6.3 - /home/jchi/.pyenv/shims/python
  Browsers:
    Chrome: 80.0.3987.87
    Firefox: 68.7.0esr
  npmGlobalPackages:
    gatsby-cli: 2.8.16
    gatsby-dev-cli: 2.5.42

[pieh]

Are you able to share your site so we can try to reproduce?

---edit
Ah I just see that you can't

[pieh]

While not exactly OOM - this issue #23221 tracks some regressions related to webpack step that might have similar cause.

Unfortunately the tracing output won't help much here, because most likely something weird is happening inside webpack compilation.

We really need to be able to reproduce this to determine the problem (in case this issue has different root cause than the issue I linked) - is there chance of sharing your site privately?

[vai0]

@pieh I'll see if I can get permission to do so. For the time being, let me know what files / areas you're interested that I can share with you until I get approval.

[pieh]

It's hard to pinpoint files any specific files.

One thing that you could try is to try is to:

• temporarily install and add @wardpeet/gatsby-plugin-sourcemap-explorer plugin to your config
• build gatsby with much more memory available first:
  node --max-old-space-size=8192 node_modules/.bin/gatsby build(you might still need to increase the value of --max-old-space-size if OOM still happens)
• share public/_analyze.html and public/_analyze.gz.html files with us (just note that this will show which npm packages and names of your frontend source files - so examine those pages before sharing them)

This would at least might tell us if this issue is connected to #23221 . That issue is already being actively worked on (we have reproduction there)

[pieh]

Can you try installing gatsby@granular-chunks canary version in your site and try to build (without those --max-old-space-size flags mentioned in previous comment)?

[vai0]

• here are the _analyze files. _analyze.html _analyze.gz.html
• building gatsby with version granular-chunks without --max-old-space-size completed successfully without OOM error.