Gatsby@2.11.0 Build failure (JavaScript heap out of memory) #15190

moonmeister · 2019-06-27T18:52:11Z

Description

Gatsby 2.11 breaks site build. I'm guessing it's the new functionality around building node_modules as well as the site (#14111).

i was updating dependencies on master when build started breaking. I've updated to 2.10.5 on an update branch with no troubles but as soon as I move Gatsby to 2.11 the build breaks.

Steps to reproduce

Clone master or update branches from https://github.com/moonmeister/moonmeister.net
update gatsby dep to gatsby@2.11.0
run build => see breakage.

Expected result

build should be successful.

Actual result

[BABEL] Note: The code generator has deoptimised the styling of /home/ajmoon/code/moonmeister/moonmeister.net/node_modules/netlify-cms-app/dist/netlify-cms-app.js as it
⠦ Building production JavaScript and CSS bundles

<--- Last few GCs --->

[19712:0x2dbca30]    45370 ms: Scavenge 1338.2 (1423.9) -> 1337.3 (1423.9) MB, 2.9 / 0.0 ms  (average mu = 0.163, current mu = 0.102) allocation failure 
[19712:0x2dbca30]    45374 ms: Scavenge 1338.4 (1423.9) -> 1337.5 (1423.9) MB, 2.8 / 0.0 ms  (average mu = 0.163, current mu = 0.102) allocation failure 
[19712:0x2dbca30]    45378 ms: Scavenge 1338.6 (1423.9) -> 1337.7 (1424.4) MB, 2.6 / 0.0 ms  (average mu = 0.163, current mu = 0.102) allocation failure 


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x34eb54adbe1d]
    1: StubFrame [pc: 0x34eb54a875c2]
Security context: 0x2caa7a21e6e9 <JSObject>
    2: /* anonymous */(aka /* anonymous */) [0x23804dd52ac9] [/home/ajmoon/code/moonmeister/moonmeister.net/node_modules/@babel/core/lib/transformation/file/merge-map.js:~155] [pc=0x34eb5534b963](this=0x01389b5022b1 <null>,m=0x1c2e6adbae29 <Object map = 0x11c6eb590b11>)
    3: arguments adaptor frame: 3->1
    4: forEach...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x8dc510 node::Abort() [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
 2: 0x8dc55c  [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
 3: 0xad9b5e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
 4: 0xad9d94 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
 5: 0xec7bf2  [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
 6: 0xec7cf8 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
 7: 0xed3dd2 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
 8: 0xed4704 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
 9: 0xed7371 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
10: 0xea07f4 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
11: 0x114018e v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
12: 0x34eb54adbe1d 
Aborted (core dumped)
error Command failed with exit code 134.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Environment

  System:
    OS: Linux 5.0 Ubuntu 19.04 (Disco Dingo)
    CPU: (8) x64 Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz
    Shell: 5.0.3 - /bin/bash
  Binaries:
    Node: 10.15.3 - ~/.nvm/versions/node/v10.15.3/bin/node
    Yarn: 1.15.2 - ~/.nvm/versions/node/v10.15.3/bin/yarn
    npm: 6.9.0 - ~/.nvm/versions/node/v10.15.3/bin/npm
  Browsers:
    Chrome: 75.0.3770.100
    Firefox: 67.0.4
  npmPackages:
    gatsby: 2.11.0 => 2.11.0 
    gatsby-image: ^2.0.34 => 2.2.3 
    gatsby-plugin-eslint: ^2.0.4 => 2.0.5 
    gatsby-plugin-manifest: ^2.0.24 => 2.2.0 
    gatsby-plugin-netlify: ^2.0.13 => 2.1.0 
    gatsby-plugin-netlify-cms: ^4.1.0 => 4.1.0 
    gatsby-plugin-offline: ^2.0.25 => 2.2.0 
    gatsby-plugin-react-helmet: ^3.0.10 => 3.1.0 
    gatsby-plugin-robots-txt: ^1.4.0 => 1.4.0 
    gatsby-plugin-sass: ^2.0.11 => 2.1.0 
    gatsby-plugin-sharp: ^2.0.30 => 2.2.1 
    gatsby-plugin-sitemap: ^2.0.11 => 2.2.0 
    gatsby-remark-images: ^3.0.10 => 3.1.2 
    gatsby-remark-relative-images: ^0.2.2 => 0.2.2 
    gatsby-source-filesystem: ^2.0.27 => 2.1.1 
    gatsby-transformer-remark: ^2.3.8 => 2.5.0 
    gatsby-transformer-sharp: ^2.1.17 => 2.2.0 
  npmGlobalPackages:
    gatsby-cli: 2.7.2
    gatsby-dev-cli: 2.5.0

The text was updated successfully, but these errors were encountered:

moonmeister · 2019-06-27T19:06:45Z

Okay, so this is definitely a node issue. It seems that the extra transpiling being done is just now making node hit it's default limit in my case. I ran export NODE_OPTIONS=--max_old_space_size=4096 to expand the defaults and build is now happy.

this seems strange that node doesn't just request more memory if it's available, also I'm a little concerned how this might affect folks build servers.

moonmeister · 2019-06-27T20:01:25Z

Yeah, affects Netlify too. I was able to get Netlify to build correctly by adding an environment variable:

[build.environment]
  NODE_OPTIONS = "--max_old_space_size=4096"

moonmeister · 2019-06-27T20:01:55Z

This seems like it might negatively affect a lot of people. Any way we can make this more memory efficient?

vickylance · 2019-06-27T20:27:08Z

@moonmeister OMG thank you so much I just updated my package.json file and suddenly this issue crept up. And yes increasing the size solved the issue.

moonmeister · 2019-06-27T20:33:25Z

@vickylance figured I wouldn't be the only one. Glad you figured it out.

vickylance · 2019-06-27T20:45:28Z

However netlify does not seem to build even after increasing the node size. Its not throwing an error but its just staying building forever.

EDIT: it got published. But it took a lot of time to build

Zellement · 2019-06-27T20:48:27Z

I've got the same issue I think, @vickylance and @moonmeister - I can test the environment variable fix if someone can guide me how? :) Not done them before - is it just as here in Netlify?

moonmeister · 2019-06-27T20:56:44Z

Not sure you need the quotes but that looks correct.

Zellement · 2019-06-27T20:57:15Z

OK - didn't work with the quotes initially. I'll try without.

EDIT: Works without quotes in Netlify

moonmeister · 2019-06-27T20:57:43Z

Yeah, build times have definitely increased. I'm not sure what value this adds, I'm sure it does but it seems to have some significant downsides too.

vickylance · 2019-06-27T20:58:41Z

Add in netlify like this.

vickylance · 2019-06-27T20:59:23Z

@moonmeister Yes my build time was less than a min and now its more than 6 mins

Zellement · 2019-06-27T21:07:07Z

Same here; environment variable fixed it, but it took an age to deploy.

Not sure about you guys but I also got a lot of lines of...

10:04:46 PM: Pkg: FLAGS_MISMATCH

Not seen that before myself.

EDIT: In Netlify
EDIT 2: Gone from 4 mins to 7 mins.

moonmeister · 2019-06-28T06:55:49Z

@gatsbyjs/core Can we get some input here? This isn't strictly a Gatsby issue but definitely caused by a decision the Gatsby project made. IS there a way to disable this for folks that don't want it, or maybe a way to optimize the process to increase performance/build times (or at least not break node)?

wardpeet · 2019-06-28T06:58:25Z

It only took 20-40s longer on gatsbyjs.org when transpiling node_modules so it's weird that it's so heavy for your sites. I'll have a look at your repro! @moonmeister

moonmeister · 2019-06-28T07:43:13Z

@wardpeet Thanks! My site is only page (aside from the cms), so it wasn't much to start with...but it probably still doubled.

If anyone else has repositories they can share that may be helpful.

homearanya · 2019-06-28T08:09:43Z

Same problem on my side. This has been driving me bananas. I'm glad to find out it's an issue on Gatsby's last version

wardpeet · 2019-06-28T09:57:29Z

Seems like gatsby-netlify-cms being the issue. I'll cook up a fix.

Sorry for the inconveniences

wardpeet · 2019-06-28T13:05:39Z

#15191 seems to be ready to get merged!

moonmeister · 2019-06-28T13:09:15Z

@wardpeet can you explain why this plugin specifically is an issue? Is it purely its size?

erquhart · 2019-06-28T14:40:33Z

gatsby-plugin-netlify-cms runs an independent Webpack build within Gatsby's Webpack build (insert "yo dawg" reference here) using a modified version of the Gatsby Webpack config. Gatsby 2.11.0 no longer excludes node_modules, which causes the plugin's Webpack build to attempt processing the Netlify CMS modules with Babel. The modules are massive and also prebuilt, so it will (and should) crash your system.

The PR I raised just adds that exclusion back in to the plugin's Webpack config. Please don't override your node max file size limit to fix this!

rwieruch · 2019-07-01T19:54:32Z

I ran into this today as well:

[BABEL] Note: The code generator has deoptimised the styling of undefined 
as it exceeds the max of 500KB.

It happens for a large markdown file (exactly this long read blog post which I want to transfer from my old Hugo website to my new Gatsby website). In Gatsby I use PrismJS and MDX. The 1600th line of markdown kills it.

So I am not entirely sure whether it is related, because I am not using Netlify. If not related, I am happy to open a new issue.

My dependencies:

  "dependencies": {
    "@mdx-js/mdx": "^1.0.21",
    "@mdx-js/react": "^1.0.21",
    "axios": "^0.19.0",
    "disqus-react": "^1.0.6",
    "dotenv": "^7.0.0",
    "gatsby": "^2.12.0",
    "gatsby-image": "^2.2.3",
    "gatsby-link": "^2.2.0",
    "gatsby-mdx": "^0.6.3",
    "gatsby-plugin-catch-links": "^2.1.0",
    "gatsby-plugin-feed": "^2.3.1",
    "gatsby-plugin-google-analytics": "^2.1.1",
    "gatsby-plugin-manifest": "^2.2.0",
    "gatsby-plugin-offline": "^2.2.0",
    "gatsby-plugin-react-helmet": "^3.1.0",
    "gatsby-plugin-sentry": "^1.0.1",
    "gatsby-plugin-sharp": "^2.2.1",
    "gatsby-plugin-sitemap": "^2.2.1",
    "gatsby-plugin-styled-components": "^3.1.0",
    "gatsby-remark-autolink-headers": "^2.1.0",
    "gatsby-remark-copy-linked-files": "^2.1.0",
    "gatsby-remark-images": "^3.1.2",
    "gatsby-remark-prismjs": "^3.3.0",
    "gatsby-source-filesystem": "^2.1.1",
    "gatsby-transformer-remark": "^2.5.0",
    "gatsby-transformer-sharp": "^2.2.0",
    "prismjs": "^1.16.0",
    "react": "next",
    "react-dom": "next",
    "react-facebook-pixel": "^0.1.1",
    "react-helmet": "~5.2.1",
    "react-quora-pixel": "0.0.5",
    "react-twitter-conversion-tracker": "^1.0.0",
    "react-youtube": "^7.9.0",
    "styled-components": "^4.3.2"
  },

Before I had Gatsby 2.3.25 and ran into the same issue (in case we want to exclude a regression bug). Then I updated full of hope that a new version would resolve it.

SpicyPete · 2019-07-02T17:10:58Z

Gatsby 2.12.x seemed to of fixed all of my issues

wernerglinka · 2019-07-03T21:08:07Z

I am following the Gatsby tutorial from https://www.netlifycms.org/docs/gatsby/
I am running Gatsby 2.13.2, build on Netlify crashed but the fix NODE_OPTIONS = "--max_old_space_size=4096" did the job.

moonmeister · 2019-07-05T00:19:38Z

@wernerglinka netlify-cms changed the name of it's package to netlify-cms-app. That may be affecting you, not sure.

jamesgrubb · 2019-07-06T13:43:35Z

Having same issue here.

erquhart · 2019-07-06T14:05:07Z

This was fixed - are you sure it's the same exact issue? Also make sure you have the latest version of both Gatsby and the Netlify CMS Gatsby plugin.

_{Sent with GitHawk}

wernerglinka · 2019-07-06T14:09:50Z

Thanks Alex, I updated to netlify-cms-app and everything works. Werner

…

Sent from my iPad

On Jul 4, 2019, at 19:20, Alex Moon ***@***.***> wrote: @wernerglinka netlify-cms changed the name of it's package to netlify-cms-app. That may be affecting you, not sure. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

jamesgrubb · 2019-07-06T15:01:39Z

I added the environment variable --max-old-space-size=4096 to netlify and managed a successful build. I deleted lock files deleted node_modules and re-installed. Same issue as far as I can `see.

"dependencies": {
    "gatsby": "^2.13.4",
    "gatsby-plugin-netlify-cms": "^4.1.1",
    "netlify-cms-app": "^2.9.5",
    "react": "^16.8.6",
    "react-dom": "^16.8.6"
  }

erquhart · 2019-07-06T15:31:17Z

Can you help us reproduce? Repo we can clone?

_{Sent with GitHawk}

rdela · 2019-07-09T07:19:55Z

In case it helps figure any of this out, this repo that uses Netlify CMS appears to still be broken under ~~2.13.6~~ 2.13.9, but everything is fine under 2.10.5. See master branch vs rdela/voidcluster#54; build failure log at ~~https://app.netlify.com/sites/voidcluster/deploys/5d20f59e9f191b000a971e08~~
https://app.netlify.com/sites/voidcluster/deploys/5d242cc7332e50bd7113ca72

@erquhart I commented on your PR already on Jul 6, 2019 at 12:40 PM PDT, a few hours after your last post here:
#15191 (comment)
...not sure whether you got notified already or not, if so, sorry for the double mention and thanks for all the work trying to resolve this!

ethan-hurst · 2019-07-10T04:49:13Z

Yeah, affects Netlify too. I was able to get Netlify to build correctly by adding an environment variable:
[build.environment]
  NODE_OPTIONS = "--max_old_space_size=4096"

This worked for me <3

rdela · 2019-07-10T06:06:08Z

Added "(JavaScript heap out of memory)" to end of issue title just now (in addition to reopening yesterday) to try and prevent more duplicate issues like #15540

wardpeet · 2019-07-10T08:44:05Z

I'll be closing this one in favor of #15256

Please move the discussion to that thread. I have created a PR that hopefully fixes the OOM issue #15588

rdela · 2019-07-11T04:07:35Z

~~#15588~~ #15591 has done the trick for me thanks @wardpeet!

vientang · 2019-08-06T02:43:47Z

I just got hit with this. Gatsby version 2.13.31, Node 10.15.3 via npx and npm 6.4.1

I had imported a file that was 8mb, which was working just a few days ago. @moonmeister 's solution worked for me - increase memory with export NODE_OPTIONS=--max_old_space_size=4096

gatsbyjs/gatsby#15190

moonmeister added type: bug An issue or pull request relating to a bug in Gatsby status: needs reproduction This issue needs a simplified reproduction of the bug for further troubleshooting. labels Jun 27, 2019

moonmeister mentioned this issue Jun 27, 2019

fix(gatsby-plugin-netlify-cms): exclude node_modules from cms #15191

Merged

moonmeister added type: bug An issue or pull request relating to a bug in Gatsby and removed type: question or discussion Issue discussing or asking a question about Gatsby labels Jun 28, 2019

wardpeet mentioned this issue Jun 28, 2019

Making babel node_modules transpilation better #15213

Closed

nikoladev mentioned this issue Jul 2, 2019

Count more reactions when considering the highest rated comment refined-github/refined-github#2190

Closed

rdela reopened this Jul 9, 2019

wardpeet mentioned this issue Jul 9, 2019

Build fail on Netlify but not locally: JavaScript heap out of memory #15540

Closed

rdela changed the title ~~Gatsby@2.11.0 Build failure~~ Gatsby@2.11.0 Build failure (JavaScript heap out of memory) Jul 10, 2019

wardpeet closed this as completed Jul 10, 2019

wKovacs64 mentioned this issue Aug 9, 2019

Build failing with heap out of memory error, gatsby@>=2.11.0 #16499

Closed

jazwiecki mentioned this issue Aug 12, 2019

fix for Netlify out-of-memory build errors jazwiecki/jimjazwiecki.com#34

Merged

edemirbag mentioned this issue Aug 27, 2019

gatsby build fails with FATAL ERROR exit code 134: JavaScript heap out of memory v4iv/gatsby-starter-business#36

Closed

vai0 mentioned this issue Apr 23, 2020

Gatsby@2.20.29 Build failure (JavaScript heap out of memory) #23402

Closed

nbw added a commit to uchidashokudo/uchida that referenced this issue May 6, 2020

Fix build issue with memory

1894048

gatsbyjs/gatsby#15190

favoyang mentioned this issue Jun 9, 2020

netlify: ineffective mark-compacts near heap limit Allocation failed openupm/openupm#726

Closed

dennishn mentioned this issue Sep 17, 2020

JavaScript heap out of memory datocms/gatsby-source-datocms#123

Closed

khavinshankar mentioned this issue Dec 15, 2022

Bump tailwindcss:3.2.4 ohcnetwork/care_fe#4307

Merged

anthonyburdi mentioned this issue Jun 15, 2023

[DOCS] Enable docs versioning for 0.16.16 great-expectations/great_expectations#8125

Merged

3 tasks

aloisklink mentioned this issue Aug 27, 2023

feat: add @mermaid-js/parser package and info langium parser mermaid-js/mermaid#4727

Merged

2 tasks

Gatsby@2.11.0 Build failure (JavaScript heap out of memory) #15190

Gatsby@2.11.0 Build failure (JavaScript heap out of memory) #15190

Comments

moonmeister commented Jun 27, 2019

Description

Steps to reproduce

Expected result

Actual result

Environment

moonmeister commented Jun 27, 2019

moonmeister commented Jun 27, 2019

moonmeister commented Jun 27, 2019

vickylance commented Jun 27, 2019

moonmeister commented Jun 27, 2019

vickylance commented Jun 27, 2019 • edited Loading

Zellement commented Jun 27, 2019

moonmeister commented Jun 27, 2019

Zellement commented Jun 27, 2019 • edited Loading

moonmeister commented Jun 27, 2019

vickylance commented Jun 27, 2019

vickylance commented Jun 27, 2019

Zellement commented Jun 27, 2019 • edited Loading

moonmeister commented Jun 28, 2019

wardpeet commented Jun 28, 2019

moonmeister commented Jun 28, 2019

homearanya commented Jun 28, 2019

wardpeet commented Jun 28, 2019

wardpeet commented Jun 28, 2019

moonmeister commented Jun 28, 2019 • edited Loading

erquhart commented Jun 28, 2019

rwieruch commented Jul 1, 2019 • edited Loading

SpicyPete commented Jul 2, 2019

wernerglinka commented Jul 3, 2019

moonmeister commented Jul 5, 2019

jamesgrubb commented Jul 6, 2019

erquhart commented Jul 6, 2019

wernerglinka commented Jul 6, 2019 via email

jamesgrubb commented Jul 6, 2019

erquhart commented Jul 6, 2019

rdela commented Jul 9, 2019 • edited Loading

ethan-hurst commented Jul 10, 2019

rdela commented Jul 10, 2019

wardpeet commented Jul 10, 2019

rdela commented Jul 11, 2019 • edited Loading

vientang commented Aug 6, 2019

vickylance commented Jun 27, 2019 •

edited

Loading

Zellement commented Jun 27, 2019 •

edited

Loading

Zellement commented Jun 27, 2019 •

edited

Loading

moonmeister commented Jun 28, 2019 •

edited

Loading

rwieruch commented Jul 1, 2019 •

edited

Loading

rdela commented Jul 9, 2019 •

edited

Loading

rdela commented Jul 11, 2019 •

edited

Loading