Gatsby develop and build hang on source and transform nodes stage on large CSV file with extremely high ram usage

[DanielPBliss]

Preliminary Checks

• This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/gatsbyjs/gatsby/issues
• This issue is not a question, feature request, RFC, or anything other than a bug report directly related to Gatsby. Please post those things in GitHub Discussions: https://github.com/gatsbyjs/gatsby/discussions

Description

When using gatsby build or gatsby develop with the plugin gatsby-transformer-csv with a largeish csv file ~16'000 rows it hangs on the 'source and transform nodes' stage seemingly indefinitely. Also the nodejs process is using approximately 50Gb out of my 64gb of memory.
Ive tried this with a new starter of gatsby 4 and created a new csv file with the same length to make sure it wasnt an issue the one project or the file was somehow corrupted/invalid.

I know a similar issue was mentioned here : #11839 (comment) but this one doesnt mention high memory usage so may have a different cause

Reproduction Link

https://github.com/DanielPBliss/GatsbySourceAndTransformIssue

Steps to Reproduce

1. Clone linked repo
2. Run 'gatsby build' or 'gatsby develop'

...

Expected Result

Build to complete and Ram usage to stay within normal range

Actual Result

Build hangs on 'source and transform nodes' stage seemingly indefinitely and ram usage quickly accumulates to about 50gb out of my 64gb.

Environment

System:
    OS: Windows 10 10.0.19042
    CPU: (8) x64 Intel(R) Core(TM) i7-7820HK CPU @ 2.90GHz 
  Binaries:
    Node: 14.15.5 - C:\Program Files\nodejs\node.EXE       
    Yarn: 1.22.5 - C:\Program Files (x86)\Yarn\bin\yarn.CMD
    npm: 6.14.11 - C:\Program Files\nodejs\npm.CMD
  Languages:
    Python: 3.8.3 - C:\Python38\python.EXE
  Browsers:
    Edge: Spartan (44.19041.1266.0), Chromium (95.0.1020.40)
  npmPackages:
    gatsby: ^4.1.0 => 4.1.0
    gatsby-plugin-gatsby-cloud: ^4.1.0 => 4.1.0
    gatsby-plugin-image: ^2.1.0 => 2.1.0
    gatsby-plugin-manifest: ^4.1.0 => 4.1.0
    gatsby-plugin-offline: ^5.1.0 => 5.1.0
    gatsby-plugin-react-helmet: ^5.1.0 => 5.1.0
    gatsby-plugin-sharp: ^4.1.0 => 4.1.0
    gatsby-source-filesystem: ^4.1.0 => 4.1.0
    gatsby-transformer-csv: ^4.1.0 => 4.1.0
    gatsby-transformer-sharp: ^4.1.0 => 4.1.0
  npmGlobalPackages:
    gatsby-cli: 3.14.2

Config Flags

No response

[nickamckenna]

"gatsby build" -> It uses 105Gb in about 30 seconds on my computer! Crazy!

[wardpeet]

My guess is that we should do csv streaming instead of loading the whole file. It's not something we will prioritize anytime soon but feel free to open a PR.

[pragmaticpat]

So @wardpeet this looks like some inefficiency within gastby-transformer-csv specifically - is that right?

[joernroeder]

I'm also experiencing very high usage of RAM on my local machine on my gatsby 4 test branch on the same build step.
Also gatsby-could (obviously) fails with Your Gatsby build's memory consumption exceeded the limits allowed in your plan. For more details, see https://gatsby.dev/memory.

I don't use the csv source plugin, but maybe the following list of yarn list | grep gatsby- can help to resolve that.

├─ custom-gatsby-plugin@1.0.0
│  ├─ gatsby-source-apiserver@^2.1.8
│  └─ gatsby-source-filesystem@^4.1.3
├─ custom-gatsby-utils@1.0.0
│  └─ gatsby-core-utils@^3.1.3
│  ├─ gatsby-background-image@^1.3.1
│  ├─ gatsby-plugin-feed@^4.1.1
│  ├─ gatsby-plugin-force-trailing-slashes@^1.0.4
│  ├─ gatsby-plugin-gatsby-cloud@^4.1.3
│  ├─ gatsby-plugin-image@^2.1.3
│  ├─ gatsby-plugin-loadable-components-ssr@^4.1.0
│  ├─ gatsby-plugin-netlify-cms@^6.1.0
│  ├─ gatsby-plugin-react-helmet-async@^1.1.1
│  ├─ gatsby-plugin-react-svg@^3.1.0
│  ├─ gatsby-plugin-robots-txt@^1.6.14
│  ├─ gatsby-plugin-sharp@^4.1.4
│  ├─ gatsby-plugin-sitemap@^5.1.1
│  ├─ gatsby-plugin-theme-ui@^0.12.0
│  ├─ gatsby-plugin-twitter@^4.1.1
│  ├─ gatsby-plugin-webpack-bundle-analyser-v2@^1.1.25
│  ├─ gatsby-remark-autolink-headers@^5.1.1
│  ├─ gatsby-remark-embedder@^5.0.0
│  ├─ gatsby-remark-responsive-iframe@^5.1.0
│  ├─ gatsby-source-datocms@^3.0.10
│  ├─ gatsby-source-filesystem@^4.1.3
│  ├─ gatsby-source-wordpress@^6.1.3
│  ├─ gatsby-transformer-cloudinary@^2.2.3
│  ├─ gatsby-transformer-json@^4.1.0
│  ├─ gatsby-transformer-remark@^5.1.4
│  ├─ gatsby-transformer-sharp@^4.1.0
│  ├─ gatsby-transformer-yaml@^4.1.0
│  └─ gatsby-core-utils@^3.1.3
│  ├─ gatsby-core-utils@^3.1.3
│  └─ gatsby-legacy-polyfills@^2.1.0
├─ gatsby-background-image@1.5.3
├─ gatsby-cli@4.1.4
│  ├─ gatsby-core-utils@^3.1.3
│  ├─ gatsby-recipes@^1.1.3
│  ├─ gatsby-telemetry@^3.1.3
├─ gatsby-core-utils@3.1.3
├─ gatsby-graphiql-explorer@2.1.0
├─ gatsby-legacy-polyfills@2.1.0
├─ gatsby-link@4.1.0
├─ gatsby-page-utils@2.1.3
│  ├─ gatsby-core-utils@^3.1.3
├─ gatsby-plugin-catch-links@4.1.0
├─ gatsby-plugin-feed@4.1.1
│  ├─ gatsby-plugin-utils@^2.1.1
├─ gatsby-plugin-force-trailing-slashes@1.0.5
├─ gatsby-plugin-gatsby-cloud@4.1.3
│  ├─ gatsby-core-utils@^3.1.3
│  ├─ gatsby-telemetry@^3.1.3
├─ gatsby-plugin-image@2.1.3
│  ├─ gatsby-core-utils@^3.1.3
├─ gatsby-plugin-loadable-components-ssr@4.1.0
├─ gatsby-plugin-netlify-cms@6.1.0
├─ gatsby-plugin-page-creator@4.1.4
│  ├─ gatsby-core-utils@^3.1.3
│  ├─ gatsby-page-utils@^2.1.3
│  ├─ gatsby-plugin-utils@^2.1.1
│  ├─ gatsby-telemetry@^3.1.3
├─ gatsby-plugin-react-helmet-async@1.2.0
├─ gatsby-plugin-react-svg@3.1.0
├─ gatsby-plugin-robots-txt@1.6.14
├─ gatsby-plugin-sharp@4.1.4
│  ├─ gatsby-core-utils@^3.1.3
│  ├─ gatsby-plugin-utils@^2.1.1
│  ├─ gatsby-telemetry@^3.1.3
├─ gatsby-plugin-sitemap@5.1.1
├─ gatsby-plugin-theme-ui@0.12.0
├─ gatsby-plugin-twitter@4.1.1
├─ gatsby-plugin-typescript@4.1.3
├─ gatsby-plugin-utils@2.1.1
├─ gatsby-plugin-webpack-bundle-analyser-v2@1.1.25
├─ gatsby-react-router-scroll@5.1.0
├─ gatsby-recipes@1.1.3
│  ├─ gatsby-core-utils@^3.1.3
│  ├─ gatsby-telemetry@^3.1.3
├─ gatsby-remark-autolink-headers@5.1.1
├─ gatsby-remark-embedder@5.0.0
├─ gatsby-remark-responsive-iframe@5.1.0
├─ gatsby-source-apiserver@2.1.8
├─ gatsby-source-datocms@3.0.10
│  ├─ gatsby-core-utils@^3.1.0
│  ├─ gatsby-core-utils@3.1.0
│  ├─ gatsby-plugin-utils@^2.0.0
│  ├─ gatsby-plugin-utils@2.0.0
├─ gatsby-source-filesystem@4.1.3
│  ├─ gatsby-core-utils@^3.1.3
├─ gatsby-source-wordpress@6.1.3
│  ├─ gatsby-core-utils@^3.1.3
│  ├─ gatsby-plugin-catch-links@^4.1.0
│  ├─ gatsby-source-filesystem@^4.1.3
├─ gatsby-telemetry@3.1.3
│  ├─ gatsby-core-utils@^3.1.3
├─ gatsby-transformer-cloudinary@2.2.4
├─ gatsby-transformer-json@4.1.0
├─ gatsby-transformer-remark@5.1.4
│  ├─ gatsby-core-utils@^3.1.3
├─ gatsby-transformer-sharp@4.1.0
├─ gatsby-transformer-yaml@4.1.0
├─ gatsby-worker@1.1.0
│  ├─ gatsby-cli@^4.1.4
│  ├─ gatsby-core-utils@^3.1.3
│  ├─ gatsby-graphiql-explorer@^2.1.0
│  ├─ gatsby-legacy-polyfills@^2.1.0
│  ├─ gatsby-link@^4.1.0
│  ├─ gatsby-plugin-page-creator@^4.1.4
│  ├─ gatsby-plugin-typescript@^4.1.3
│  ├─ gatsby-plugin-utils@^2.1.1
│  ├─ gatsby-react-router-scroll@^5.1.0
│  ├─ gatsby-telemetry@^3.1.3
│  ├─ gatsby-worker@^1.1.0

[joernroeder]

shot in the dark, but it could be the same issue #34081

[github-actions]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[witcradg]

I guess I need to fork the entire gatsby project to create the PR for this. @joernroeder

[witcradg]

I've committed the change to my fork. I'm not sure I'm following the right procedure to hand it off but you can find it here:
https://github.com/witcradg/gatsby
@joernroeder
If I haven't done this as a proper PR, let me know and I'll do it again.

[github-actions]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[LekoArts]

@witcradg We'd be happy to receive this as a PR. Feel free to send it in! :)

[smf706fish]

Any updates on this??? Having a similar problem trying to build a gatsby website with 10,000+ pages from 50 csv files for each state and then 1 big csv file with all url slugs in it. Site builds using about 1/6 of the data, building with anymore and gets stuck on "source and transform nodes" then stops with a "Killed: 9" message. Tried --max-old-space-size=8192 as well as fix @witcradg made with csv transformer and no luck. Also tried breaking the url slug csv into smaller csvs with no luck.

[joernroeder]

I might be able to create a pull request for this over the weekend. Just need the time to look into it.

[AshuuDixit]

I might be able to create a pull request for this over the weekend. Just need the time to look into it.

[github-actions]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[jidicula]

I get a trace from dmesg saying

CODE SIGNING: cs_invalid_page(0x11635c000): p=14585[node] final status 0x23010204, denying page sending SIGKILL

when I run gatsby develop or gatsby build. The killed message shows up while plugins are loading.

❯ npx gatsby build --verbose
verbose set gatsby_log_level: "verbose"
verbose set gatsby_executing_command: "build"
verbose loading local command from:
path/to/project/node_modules/gatsby/dist/commands/build.js
verbose running command: build
success compile gatsby files - 0.192s
success load gatsby config - 0.036s
⠋ load plugins
[1]    15593 killed     npx gatsby build --verbose

[github-actions]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[github-actions]

Hey again!

It’s been 60 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.
Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to comment on this issue or create a new one if you need anything else.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community! 💪💜