Heap memory usage increase during development builds after upgrading to v4.24.5 #36899
Comments
skatingincentralpark
added
the
type: bug
gatsbot
bot
added
the
status: triage needed
LekoArts
added
status: confirmed
|
Hi, thanks for the issue! Also thanks for the detailed issue. I suppose you updated your app from 4.15 to 4.24 without any incremental updates right?
If then e.g. 4.17 would have it, double check that 4.16 is ok. Then one could be sure that the 4.17 release has this regression. If you have the time, we'd much appreciate it if you (or anyone else) could help with this. |
|
Hey @LekoArts, here are the results from the investigation (credits: @odunet ⭐️)! We've also recorded the PartyTown Context error versions, but that can be ignored as the latest versions have implemented the fix. Gatsby versions investigation results 'v4.15.0 - 'v4.24.4'
As shown above, Gatsby v4.16.0 in commit a88703f introduced a PartyTown Context bug, this bug rendered versions v4.16.0 - v4.19.0 unusable for us. This was later fixed in Gatsby v4.20.0 in commit 49cf094. It can also be deduced from above that the OOM (Out of memory) error was introduced in Gatsby v4.19.0, the OOM error was accompanied by an increase of approximately 9GB in the webpack 'pack' file size. Thank you ✊ |
|
Awesome, thank you for help on this! We'll have a closer look at this next week 👍 |
|
Any updates on this? Seems like gatsby 5 is also affected Can confirm it works well with 4.15.2 |
LekoArts
added
status: needs more info
|
I've re-ran your reproduction again and I don't see the behavior described anymore. I think I saw it when I initially tried it but that could have been just a coincidance. 4.15.2:
4.24.5
As you can see the heap is really similar. And this also makes sense. Node.js will use all the available memory and only then garbage collect. We then tried to force garbage collection and also only saw 350 MB vs 400MB. You can try that for yourself: // in your gatsby-node.js
// ...rest of imports
const v8 = require(`v8`)
const vm = require(`vm`)
v8.setFlagsFromString(`--expose_gc`)
const gc = vm.runInNewContext(`gc`)
exports.createPages = async ({ graphql, actions, reporter }) => {
setInterval(() => {
// Prints out a string containing stats about your Node.js process.
gc()
console.log(top.toString())
}, 5000)
// ... rest of filesSo the important question is: Did you get out of memory errors? Because if not then this behavior isn't too unusual. |
|
Hey @LekoArts, thank you very much for your response. I just re-ran the reproduction (following the steps outlined in the issue) on two computers multiple times and am still seeing the behaviour described in the issue (which differs from your results). In answer to your question "Did you get out of memory errors?": Yes we do get OOM errors in our main application of the same structure (since it is much larger in size). Which prevents us from starting our dev server.
I've created a reproduction of the OOM error in a new branch. Instructions are at the bottom of the comment Your heap usage with forced garbage collection seems to align with my results. Questions
Summary of results
Screenshots of results and System InfoMacbook Pro 13"
Macbook Pro 14"
Macbook Pro 13" - Garbage Collected
Steps to produce OOM errorI have created a branch on the reproduction repo that will have the OOM error. Below are the steps.
|
|
We started running into OOM issues as well, I will also post some of the testing we did in a bit. @skatingincentralpark maybe a workaround for you (or something to test/benchmark as well) is with the |
|
Awesome, Handy to have in the back pocket though. |
|
the heap memory issue remained in gatsby5, the scope: the raised problems:
I hope this comment be helpful, for those who wants to upgrade to reconsider their strategy, |
|
I have a problem with memory leak for a long time. I wait a fix and don't update my projects from 4.14.1 to newest versions. |
|
I have this issue too after updating from Gatsby 4.14.1 to Gatsby 5 (and to react-plugin-mdx). My site is pretty big, with around 1,000 pages and 17,000 images. None of the workarounds work and I can't build my site. I have tried adjusting the heap limit to 16gb with
edit: Managed to get my build to finish by closing all background apps and raising the --max-old-space-size to 24gb (at its peak it used 19gb during build). This also helps gatsby develop to not crash as frequently. Overall memory usage is now 27/32gb when running gatsby develop, so this obviously isn't a practical solution for the majority of people who will have 16GB of RAM or less, and attempting to load certain pages consumes all system memory and crashes it. Sidenote: My builds also seem to take A LOT longer now than before. On Gatsby 4 the first build took ages, but subsequent builds took less (like maybe half an hour, and most of that was generating images). On Gatsby 5 my first successful build took around 5 hours, and subsequent builds also took 5 hours. It was processing images it had already processed the first time. It seems like caching is broken? |
|
Hiya! This issue has gone quiet. Spooky quiet. 👻 We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here. Thanks for being a part of the Gatsby community! 💪💜 |
github-actions
bot
added
the
stale?
|
Any updates? |
github-actions
bot
removed
the
stale?
|
Is there anything that community(we) can help with?, |
I don't use gatsby-plugin-mdx and gatsby-plugin-material-ui in my projects, but I still have heap memory issues. |
|
Hi, I commented before on this thread here and here. the steps that we followed is as below:
important note on running development server: |
|
We haven't tried the updated reproduction since my last comment, but already back then we couldn't verify what you were seeing unfortunately. The issue is also now a mix between "just" increased memory usage and OOM. It's not clear to us what the actual problem is now. Anyways, as per https://www.gatsbyjs.com/docs/reference/release-notes/gatsby-version-support/ I'm afraid that we can't invest a lot of time chasing this down in the So my asks are:
Thanks! |
|
this is happening to us on gatsby 5.4.2 |
|
I'm having this problem with version 5.4.2 Before making these customizations to fallback: {
fs: false,
constants: false,
http: require.resolve('stream-http'),
https: require.resolve('https-browserify'),
os: require.resolve('os-browserify/browser'),
tty: require.resolve('tty-browserify'),
+ path: require.resolve('path-browserify'),
+ stream: require.resolve('stream-browserify'),
+ crypto: require.resolve('crypto-browserify'),
+ zlib: require.resolve('browserify-zlib'),
+ buffer: require.resolve('buffer/'),
},
};+ plugins: [
+ new webpack.ProvidePlugin({
+ Buffer: ['buffer', 'Buffer'],
+ }),
+ new webpack.ProvidePlugin({
+ process: 'process/browser',
+ }),
+ ],Now I'm having the same problem described in this issue. What could be happening? |
@sibelius Please understand that these types of comments are not helpful for us as they are not actionable. As mentioned in #36899 (comment) we weren't able to reproduce this issue yet. Also "this" is quite ambiguous, what do you mean with "this"? We want to help y'all but we need a https://www.gatsbyjs.com/contributing/how-to-make-a-reproducible-test-case/, actionable comments (no "+1" comments) and more details. |
|
we got this "fixed" like this giving 4GB for gatsby build |
Sorry, but it is not helpful. The build shouldn't consume 2.5gb-3gb in version 4.14.1 and 4-10gb in new versions. |
|
Note: This response comes from the team that created this issue. After lots of testing we were able to fix our issue and successfully upgrade to Gatsby v5.4.2 with the following changes. Early Tests
Conclusion
❌ Before
The JavaScript heap out of memory error would trigger when memory exceeded 5GB even with
✅ After
gatsby-config.js {
resolve: 'gatsby-plugin-emotion',
options: {
sourceMap: false,
},
},Instead of the memory heap climbing above 4GB it garbage collects at 4GB and dip back to a reasonable range.
|
|
Using your repo I was able to narrow things down to introduction of Thanks! |
LekoArts
added
status: confirmed
@mjBayati I'm interested in several of the things you did here to mitigate your issues. This is the most unique advice I've seen on build performance with Gatsby, but some of the things you did are a little hard to determine how to replicate.
Thanks! |
|
@pixelsoup thank you very much for the detailed reports and your conclusion. Your conclusions (disabling all source map generation, granting more memory and some other optimizations) led us to a 50-60% decrease now but I would not really call that a "fix" because the memory requirements are still significantly higher than before, it's just bringing it back into a state that works at all. We are using gatsby 5.5, MDX v2 and emotion and are still struggling with peak memory issues in the build step. The issue likely came up with one of the 4.x releases in our case and got amplified by upgrading MDX to v2 now, causing this investigation. It's small sites in terms of number of pages (< 200) but a nontrivial gatsby usage. Specifically, the issue is happening in the "Building HTML renderer" step of the build and it has the sneaky property of the RAM only spiking up for less than a second if run on a cached build. I'm forcing GC now, too before logging but it's clear that the memory report is not only uncollected thrash but actual memory assignments. I'm aware that by title this discussion is about issues in the development server, but I found that analyzing the problem via the build process is more predictable and better to debug. Once we got our users productive again through a mix of throwing hardware at the problem and the above mentioned optimizations we'll try to get to a reproduction scenario on a current gatsby version. May take a bit though. UPDATE: the build time issue turned out unrelated (but an actual issue rooted in the architectural changes in the MDX v2 plugin). |
|
Hey folks, I located a reason for increased memory usage (at least in repro provided by @pixelsoup ) and put up draft PR ( #37619 ) to address it somewhat (i.e. if you don't use Head API, you should get back to pre 4.19 memory usage levels). This allowed me to go from (heap max - to (heap max - ( That won't help at all with build problem (mentioned by @nkuehn ), but is at least a start at addressing these problems |
|
UPDATE: the build time issue turned out to be unrelated (MDX v2 plugin issue, found a workaround). @pieh thank you for helping out! We do have a huge memory issue in the development mode, too. I just turned to debugging the build first because it was more reproducible. Caveat: I can now say the build and develop time issues are unrelated. But I can also confirm the develop time issue discussed here. EDIT / UPDATE: (removed, turned out unrelated to this discussion) |
Hi, thanks for your deep diving into my comment,
and also what we did was removing concurrent builds and ** Also it's important to separate your by this kind of works, we was able to make our dev and production servers runnable!. |
|
An indication to whomever may be competent to investigate the issue better: Development mode with the discussed heap issues (< 50 pages project) creates an 11GB sized |
|
I released
This won't fully address this, but after few test runs I saw that using this snippet in exports.onCreateWebpackConfig = ({ actions, stage, getConfig }) => {
if (stage === `develop` || stage === `develop-html`) {
actions.setWebpackConfig({
cache: {
compression: "gzip",
},
})
}
}keeps memory usage lower than default with no compression at all (it also does decrease cache files size on disk, but I don't think that was main issue and rather just symptom). You can also try go even further and disable webpack caching completely which decreases memory usage in my tests quite a bit (but makes dev server restarts longer, so trade-off, but might be waranted to use for now if dev server just crashes on OOMs): exports.onCreateWebpackConfig = ({ actions, stage, getConfig }) => {
if (stage === `develop` || stage === `develop-html`) {
actions.setWebpackConfig({
cache: false,
})
}
} |
|
@pieh thank you! Here's my results in a test site with our custom theme and ca 150 MDX pages across two templates, tracked as recommended above with a forced garbage collection before the "process-top" call.
In any case, it takes a very long time until the first page is loaded in the dev preview but from then on things are fast - time that gets longer with the number of pages but only in the bundle generation stage. Since you pointed to the cache settings I've tried out various combinations of webpack cache configuration and found the following to at least freeing a lot of the memory again after navigating to a second page in the dev server (not on the first page view though and only if the idleTimeout: 100, // defaults to 60000 millis
idleTimeoutAfterLargeChanges: 100, // defaults to 1000 millis
allowCollectingMemory: true, // defaults to false.
maxMemoryGenerations: 0, // see docs https://webpack.js.org/configuration/cache/#cachemaxmemorygenerationsTL:Dr: Improvements but nothing that looks like it's addressing a root cause. |
|
For develop specifically we hope #37040 (once we are able to resume work on it) could help a lot for larger sites as webpack would just skip compilatation for a lot of modules that are not needed for what the user need to display a single page on dev server. I've been doing some memory snapshoting / allocation timeline stuff and I definitely saw a lot of memory being allocated specifically during serialization (part of webpack's caching pipeline). I will continue to dig into the entire pipeline for MDX overall (not just develop), but I don't have anything concrete yet on it. |
|
@pieh thanks! concerning the MDX build memory issue my current hack is to import the template components that are used in many pages into |
|
Any updates, please? The builds are still failing for Gatsby 5.8.0 |
|
Hi All, Thanks to everyone who provided comments and experiences in troubleshooting the memory issues for both development and production builds. I wanted to add another data point to the discussion. This open source project is using Gatsby v.4.25.4, and the desire was to move to gatsby-plugin-mdx v4 (and MDX v2) to resolve app bundle bloat issue and bring down the page weight. Before attempting MDX v2 migration, the site had no apparent memory issues on Gatsby v.4.25.4. After making all the migration changes, the production builds kept resulting in OOM heap memory errors during the What worked for me regarding the Production Build (so far?) As many noted, I was able to successfully execute a production build, by adding memory to the process with the command: I tried many suggestions, and the only thing that worked for me was @nkuehn's gatsby-ssr workaround.
I imported everything used as a shortcode and the template files. I declared an array variable that included all of them. The project is open source and you can see the file itself here, if not, it is here in the dropdown. gatsby-ssr.jsbuild logs: Cold production build - gatsby-plugin-mdx v3 | MDX v1 - any heap memory usage over 1.5MB is logged: Success (Done building in 978.840267548 sec)Cold production build - gatsby-plugin-mdx v4 | MDX v2 - no gatsby-ssr.js workaround - any heap memory usage over 1.5MB is logged: Heap Memory ErrorCold production build - gatsby-plugin-mdx v4 | MDX v2 - with gatsby-ssr.js workaround - any heap memory usage over 1.5MB is logged: Success (Done building in 1035.176506423 sec)After rejoicing about the production build working, I ran into issues with the development builds where it would successfully build (but take a long time), but was difficult to browse pages with the following:
What worked for me regarding the Development Build (so far?) I followed @mjBayati comment to try to limit the mdx files needed for the development builds by using the limit mdx filesI also installed With those two things active, I could navigate my development build and it seemed the file sizes and memory usage was stable at approx. 4-5GB. The development builds consistently finished (logs below for before and after the migration). The issues with climbing memory and file sizes happened after the develop build was complete and there was no limitation on the number of mdx files. build logs: Cold development build - gatsby-plugin-mdx v3 | MDX v1 - any heap memory usage over 1.5MB is logged (Building development bundle - 209.777s):Cold development build - gatsby-plugin-mdx v4 | MDX v2 - no gatsby-ssr.js workaround - any heap memory usage over 1.5MB is logged (Building development bundle - 708.665s):Cold development build - gatsby-plugin-mdx v4 | MDX v2 - with gatsby-ssr.js workaround - any heap memory usage over 1.5MB is logged: Success (Building development bundle - 822.881s)notes: Gatsby Info
Logging tools usedI used the following tools while investigating the OOM issue:
process-top code
I did not try the tools described here. Reference Documents
Cheers, |
|
Hi Everyone, The below was done based on the suggestion in this thread that the rewritten In an effort to get a development build working without limiting the number of mdx files (as in my previous post), I saw some progress in making the unrestricted development builds functional by reducing the file size/bundle of pages created from mdx files
For example, the project has an integration template that is used by over 200 mdx files, and each of these mdx files included the same two images that had a combined file size of about 2MBs. I resized and optimized these images to less than a combined size of 200kbs. This action reduced the amount of memory used by node on my machine. I also found a lightbox library being imported in some of the templates that was 300kbs large. Removing that library further reduced the memory used by node.
Lastly, since the project is on React 17 and does not have access to lazy loading in React 18, I used Maybe those who have hundred or thousands of mdx files without memory issues in their builds are able to keep the resulting page small in file size? A couple of side notes
I hope this data point is helpful and others can share their own regarding this issue. Any feedback, analysis, or suggestion is very welcomed, I have no experience in node memory issues and happy to learn. Cheers, |
|
in my case it uses about 20+gb at peak while on previous versions it used no more than 6gb: https://github.com/vsolanogo/apipka that huge amount of ram killed project and there're no way to figure out what caused it |
|
I've been running into out-of-memory issues in Gatsby since I started using it. I have a documentation tool that generates different kinds of output, and one of those is a Gatsby site. It's possible I am doing something naive in terms of layout etc., but so far my attempts to optimize the configuration haven't worked out. I did a few things to change the webpack chunking but it seems like Gatsby might be overriding the settings. So what I have done is auto-generate some fake documentation and used my tool to generate the site for it. I've checked that into this repository: https://github.com/orval/gatsbyrakoshtest There is a I'm hoping that this test case will help the maintainers look into this issue. Or at least perhaps point out something that I'm doing that is horribly wrong! [edit] This branch has 577 MDX files totalling 3.1Mb and blows 4Gb of heap: https://github.com/orval/gatsbyrakoshtest/tree/577_mdx Webpack analyser output:
|
Preliminary Checks
Description
We have a large app in a monorepo using Gatsby v4.15.2. Upon upgrading to v4.24.5, and running
gatsby developcauses the build to crash during Building development bundle with a Javascript heap out of memory error. See error snapshot below:Error Snapshot - Gatsby v4.24.5 development build
Comparison between 'v4.15.2', and 'v4.24.5' dev build memory consumption
For this test, we did the following:
NODE_OPTIONS=--max-old-space-size=8192to our dev scriptOur Gatsby 'v4.24.5' build was now successful due to the increase in allocated heap memory from step (2) above. Also, we noticed an increase of ~2GB when comparing heap memory usage for Gatsby v4.15.2, and Gatsby v4.24.5, see table below:
Reproduction Link
https://github.com/skatingincentralpark/gatsby-v4-test
Steps to Reproduce
/gatsby-v4-testgit switch 4.15.2-no-packagesto switch to the branch with Gatsby v4.15.2yarnto install dependenciesyarn devto start development serveryarn cleanto clear Gatsby cachegit switch 4.24.5-no-packagesto switch to the branch with Gatsby v4.24.5yarnto install dependenciesyarn devto start development serverExpected Result
Expected the heap usage to be similar after upgrading to v4.24.5 from v4.15.2.
Actual Result
Heap usage increased by 1GB. This increased memory usage in Gatbsy v4.24 causes a 'Javascript heap out of memory' error if
NODE_OPTIONS=--max-old-space-size=8192is not added to our dev script.Environment
System: OS: macOS 11.6 CPU: (8) arm64 Apple M1 Shell: 5.8 - /bin/zsh Binaries: Node: 14.17.6 - ~/.nvm/versions/node/v14.17.6/bin/node Yarn: 1.22.18 - ~/.nvm/versions/node/v14.17.6/bin/yarn npm: 6.14.15 - ~/.nvm/versions/node/v14.17.6/bin/npm Languages: Python: 2.7.16 - /usr/bin/python Browsers: Chrome: 106.0.5249.119 Firefox: 104.0.2 Safari: 15.0 npmGlobalPackages: gatsby-cli: 4.10.1Config Flags
No response
The text was updated successfully, but these errors were encountered: