feat(gatsby): Optimize creating many child nodes from one parent #35504

KyleAMathews · 2022-04-27T18:56:03Z

Description

A user migrating from v3 to v4 said their json transforming got a lot lot slower — and provided this test repo https://github.com/pogo19/gatsby4-slow-json

Creating nodes was always going to get slower w/ the introduction of LMDB but the slowdown seemed far more than expected.

Investigating, I saw a couple of issues.

Each time a new child node is created, the parent node is updated with the id of the new child node. When we were replacing objects in memory, this was fast enough but gets a lot slower when writing to LMDB. A large JSON file transformed could mean 1000s of writes for a parent node. To fix this, this PR adds a batcher for updating the parent node which reduces the writes by a 1000x.
Also in transformer-json specifically we're awaiting each createNode — introduced in fix(gatsby-transformer-json): Fix high memory consumption #34084. This was done to reduce memory pressure introduced by writes to lmdb queuing up in memory due to the event loop being blocked due to sync createNode calls. While this fixed the memory issue, it makes writing a lot slower than it needs to as it waits for LMDB to flush each new node to disk — we can have our cake and eat it too by batching sync createNode calls and then unblocking the event loop for the write to finish (clearing out memory).

This speeds up creating nodes by ~3.5x (~36s -> 10s roughly on the test repo).

I also added a type cache for generating type names which saves roughly 1s / 10k nodes.

Additional changes

The PR #36610 added the same optimization as #34084 to the CSV transformer, so this PR needs to handle this, too.

pogo19 · 2022-05-03T11:06:50Z

I have updated the test repo https://github.com/pogo19/gatsby4-slow-json to reflect more the real-world usage.

With "gatsby" package in latest 3.x version the "source and transform nodes" phase takes less than 10 seconds (after clean). With 4.x the process gets killed by OS ... in real-world webapp it doesn't happen (probably a bit different data entries' size, had to change it due to some data anonymization), even though data files are even bigger in real-world usage. In real-world app it takes approx 800-1200 seconds. All times are on my development notebook (4yo i7 with 16GB RAM and SSD).

pogo19 · 2022-05-03T11:11:04Z

Also as mentioned in discord thread https://discord.com/channels/484383807575687178/966709737968250910 ...

When testing this PR there were also differences in reporting and result:

Number of node types: 10. Nodes per type: Directory: 1, File: 4, NodeJson: 3926, NodeRevisionsJson: 5602, Site: 1, SiteBuildMetadata: 1, SitePage: 977, SitePlugin: 41, UrlAliasJson: 8814, UsersJson: 23
(original transformer-json) vs (pull-request)
Number of node types: 10. Nodes per type: Directory: 1, File: 4, NodeJson: 100, NodeRevisionsJson: 100, Site: 1, SiteBuildMetadata: 1, SitePage: 50, SitePlugin: 41, UrlAliasJson: 100, UsersJson: 23

Also the build with the updated transformer-json failed later (with released version of tranformer-json it worked) : There was an error in your GraphQL query: Variable "$nid" of type "String!" used in position expecting type "Date".

KyleAMathews · 2022-05-31T23:54:02Z

Ok, I fixed creating nodes so now the right number of nodes are being created and w/o memory problems. On my machine it's taking ~184s to create the nodes now.

But I discovered something very interesting. If you disable the createParentChildLink call then sourcing drops from 184s to just 6s!

The reason is that when createPageChildLink is called, it rewrites the parent File node's children field to include a link to the newly created child node and then this gets written out again to LMDB. But the problem here is this happens over and over and over again for each new child node. Debouncing means we avoid 1000s of extra writes to the db.

I tried debouncing this write in 7ceceea and that kept us around 6.5-7s for a cool ~25x speedup.

…rray is updated at least once before schema creation

KyleAMathews · 2022-06-01T01:48:36Z

Failing tests means the key for debouncing needs to be both the node id & type of the child.

KyleAMathews · 2022-06-01T05:28:53Z

We need to guarantee that the parent node is written when sourcing is finished. Still pondering best way to do that. Any ideas appreciated!

packages/gatsby/src/datastore/lmdb/lmdb-datastore.ts

pogo19 · 2022-06-02T23:34:23Z

Yeah, works like a charm. Thanks!

In my real-world app the "source and transform nodes" phase takes 8.732s now on my notebook, so it is on par with 3.x. Also no other problems and the app works OK.

Thanks again! Looking forward to a release with this merged ;)

KyleAMathews · 2022-06-03T00:47:10Z

Great! Thanks for continuing to test! Always best to get more real-world validation.

pogo19 · 2022-07-06T19:04:19Z

Is there any chance to get this merged and released? Can I help in any way?

pogo19 · 2022-09-13T11:36:42Z

Any news?

psykzz · 2022-09-17T18:53:35Z

Is it possible to have a beta version of these released that could make testing easier?
I believe i'm having the same problem with complex data types, and would love to validate.

KyleAMathews · 2022-10-12T20:52:25Z

Sorry about the delay here — you can try out this PR by installing gatsby@alpha-transformer-json

psykzz · 2022-10-12T22:46:36Z

Can confirm this has shown massive improvement in my case.
With gatsby@next I saw success source and transform nodes - 215.865s
With gatsby@alpha-transformer-json i saw success source and transform nodes - 14.541s.

Setup

I have a number json files (one per locale) that is similar in format to this file, but around 80mb.

Logs

`gatsty@next`

success load gatsby config - 0.069s
warn Plugin gatsby-transformer-json is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.0.0-next
warn Plugin @sentry/gatsby is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^2.0.0 || ^3.0.0 || ^4.0.0
warn Plugin gatsby-source-filesystem is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.0.0-next
warn Plugin gatsby-source-filesystem is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.0.0-next
warn Plugin gatsby-source-filesystem is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.0.0-next
warn Plugin gatsby-plugin-styled-components is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.0.0-next
warn Plugin gatsby-source-filesystem is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.0.0-next
warn Plugin gatsby-plugin-react-i18next is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.16.0
warn Plugin gatsby-plugin-newrelic is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@3.x || 4.x
warn Plugin gatsby-plugin-react-helmet is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.0.0-next
warn Plugin gatsby-source-filesystem is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.0.0-next
warn Plugin gatsby-plugin-image is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.0.0-next
warn Plugin gatsby-transformer-sharp is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.0.0-next
warn Plugin gatsby-plugin-sharp is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.0.0-next
warn Plugin gatsby-plugin-manifest is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.0.0-next
warn Plugin gatsby-plugin-netlify is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.0.0
warn Plugin gatsby-transformer-json is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.0.0-next
warn Plugin @sentry/gatsby is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^2.0.0 || ^3.0.0 || ^4.0.0
warn Plugin gatsby-source-filesystem is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.0.0-next
warn Plugin gatsby-source-filesystem is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.0.0-next
warn Plugin gatsby-source-filesystem is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.0.0-next
warn Plugin gatsby-plugin-styled-components is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.0.0-next
warn Plugin gatsby-source-filesystem is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.0.0-next
warn Plugin gatsby-plugin-react-i18next is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.16.0
warn Plugin gatsby-plugin-newrelic is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@3.x || 4.x
warn Plugin gatsby-plugin-react-helmet is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.0.0-next
warn Plugin gatsby-source-filesystem is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.0.0-next
warn Plugin gatsby-plugin-image is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.0.0-next
warn Plugin gatsby-transformer-sharp is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.0.0-next
warn Plugin gatsby-plugin-sharp is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.0.0-next
warn Plugin gatsby-plugin-manifest is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.0.0-next
warn Plugin gatsby-plugin-netlify is not compatible with your gatsby version 5.0.0-next.0 - It requires gatsby@^4.0.0
success load plugins - 1.372s
warn gatsby-plugin-react-helmet: Gatsby now has built-in support for modifying the document head. Learn more at https://gatsby.dev/gatsby-head
success onPreInit - 0.007s
info One or more of your plugins have changed since the last time you ran Gatsby. As
a precaution, we're deleting your site's cache to ensure there's no stale data.
success initialize cache - 0.215s
success copy gatsby files - 1.310s
success Compiling Gatsby Functions - 0.352s
success onPreBootstrap - 0.398s
success createSchemaCustomization - 0.020s
success gatsby-plugin-react-i18next: create node: el-gr/translation - 0.143s
success gatsby-plugin-react-i18next: create node: de-de/translation - 0.165s
success gatsby-plugin-react-i18next: create node: cs-cz/translation - 0.180s
success gatsby-plugin-react-i18next: create node: es-mx/translation - 0.195s
success gatsby-plugin-react-i18next: create node: it-it/translation - 0.212s
success gatsby-plugin-react-i18next: create node: en-us/translation - 0.227s
success gatsby-plugin-react-i18next: create node: es-ar/translation - 0.240s
success gatsby-plugin-react-i18next: create node: id-id/translation - 0.250s
success gatsby-plugin-react-i18next: create node: ja-jp/translation - 0.257s
success gatsby-plugin-react-i18next: create node: pt-br/translation - 0.264s
success gatsby-plugin-react-i18next: create node: ms-my/translation - 0.268s
success gatsby-plugin-react-i18next: create node: pl-pl/translation - 0.272s
success gatsby-plugin-react-i18next: create node: ko-kr/translation - 0.278s
success gatsby-plugin-react-i18next: create node: es-es/translation - 0.284s
success gatsby-plugin-react-i18next: create node: ro-ro/translation - 0.286s
success gatsby-plugin-react-i18next: create node: fr-fr/translation - 0.288s
success gatsby-plugin-react-i18next: create node: ru-ru/translation - 0.292s
success gatsby-plugin-react-i18next: create node: hu-hu/translation - 0.291s
success gatsby-plugin-react-i18next: create node: vn-vn/translation - 0.290s
success gatsby-plugin-react-i18next: create node: th-th/translation - 0.287s
success gatsby-plugin-react-i18next: create node: zh-tw/translation - 0.284s
success gatsby-plugin-react-i18next: create node: zh-my/translation - 0.277s
success Checking for changed pages - 0.001s
success source and transform nodes - 215.865s
success building schema - 0.864s
success Building champion pages - 1.630s
success Building champion skin pages - 8.641s
success Building league of legends pages - 8.656s
success Building Creator redirects - 0.018s
success createPages - 10.340s
warn Warn: updating default matchPath for all pages /en-us/404/
success createPagesStatefully - 0.133s
info Total nodes: 66570, SitePage nodes: 44115 (use --verbose for breakdown)

`gatsby@alpha-transformer-json`

success load gatsby config - 0.066s
warn Plugin gatsby-plugin-react-i18next is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^4.16.0
warn Plugin gatsby-plugin-react-i18next is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^4.16.0
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-typescript is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
warn Plugin gatsby-plugin-page-creator is not compatible with your gatsby version 4.14.0-alpha-transformer-json.27+ed04dda231 - It requires gatsby@^5.0.0-alpha-v5
success load plugins - 1.070s
warn gatsby-plugin-react-helmet: Gatsby now has built-in support for modifying the document head. Learn more at https://gatsby.dev/gatsby-head
success onPreInit - 0.007s
info One or more of your plugins have changed since the last time you ran Gatsby. As
a precaution, we're deleting your site's cache to ensure there's no stale data.
success initialize cache - 0.173s
success copy gatsby files - 1.024s
success Compiling Gatsby Functions - 0.271s
success onPreBootstrap - 0.323s
success createSchemaCustomization - 0.025s
success gatsby-plugin-react-i18next: create node: cs-cz/translation - 0.112s
success gatsby-plugin-react-i18next: create node: de-de/translation - 0.133s
success gatsby-plugin-react-i18next: create node: en-us/translation - 0.145s
success gatsby-plugin-react-i18next: create node: es-ar/translation - 0.158s
success gatsby-plugin-react-i18next: create node: es-es/translation - 0.174s
success gatsby-plugin-react-i18next: create node: el-gr/translation - 0.189s
success gatsby-plugin-react-i18next: create node: es-mx/translation - 0.200s
success gatsby-plugin-react-i18next: create node: fr-fr/translation - 0.206s
success gatsby-plugin-react-i18next: create node: ja-jp/translation - 0.215s
success gatsby-plugin-react-i18next: create node: id-id/translation - 0.222s
success gatsby-plugin-react-i18next: create node: hu-hu/translation - 0.227s
success gatsby-plugin-react-i18next: create node: ko-kr/translation - 0.233s
success gatsby-plugin-react-i18next: create node: ms-my/translation - 0.238s
success gatsby-plugin-react-i18next: create node: vn-vn/translation - 0.241s
success gatsby-plugin-react-i18next: create node: ro-ro/translation - 0.249s
success gatsby-plugin-react-i18next: create node: tr-tr/translation - 0.254s
success gatsby-plugin-react-i18next: create node: zh-tw/translation - 0.256s
success gatsby-plugin-react-i18next: create node: zh-my/translation - 0.259s
success gatsby-plugin-react-i18next: create node: pl-pl/translation - 0.260s
success gatsby-plugin-react-i18next: create node: th-th/translation - 0.264s
success gatsby-plugin-react-i18next: create node: ru-ru/translation - 0.263s
success gatsby-plugin-react-i18next: create node: pt-br/translation - 0.262s
success gatsby-plugin-react-i18next: create node: it-it/translation - 0.261s
success Checking for changed pages - 0.002s
success source and transform nodes - 14.541s
success building schema - 0.824s
success Building champion pages - 1.554s
success Building champion skin pages - 8.650s
success Building league of legends pages - 8.665s
success Building Creator redirects - 0.018s
success createPages - 10.346s
warn Warn: updating default matchPath for all pages /en-us/404/
success createPagesStatefully - 0.133s
info Total nodes: 22455, SitePage nodes: 44115 (use --verbose for breakdown)

psykzz · 2022-10-12T22:53:07Z

I also rolled back to the current gatsby@4 version i'm running to provide the timing increase from v4 -> v5.
spoilers, nothing changed between v4/v5.

gatsby@4

``` success load gatsby config - 0.060s success load plugins - 1.119s warn gatsby-plugin-react-helmet: Gatsby now has built-in support for modifying the document head. Learn more at https://gatsby.dev/gatsby-head success onPreInit - 0.011s success initialize cache - 0.183s success copy gatsby files - 0.944s success Compiling Gatsby Functions - 0.289s success onPreBootstrap - 0.331s success createSchemaCustomization - 0.020s success gatsby-plugin-react-i18next: create node: de-de/translation - 0.122s success gatsby-plugin-react-i18next: create node: en-us/translation - 0.137s success gatsby-plugin-react-i18next: create node: es-ar/translation - 0.151s success gatsby-plugin-react-i18next: create node: el-gr/translation - 0.165s success gatsby-plugin-react-i18next: create node: es-es/translation - 0.177s success gatsby-plugin-react-i18next: create node: es-mx/translation - 0.190s success gatsby-plugin-react-i18next: create node: cs-cz/translation - 0.201s success gatsby-plugin-react-i18next: create node: it-it/translation - 0.209s success gatsby-plugin-react-i18next: create node: ja-jp/translation - 0.215s success gatsby-plugin-react-i18next: create node: fr-fr/translation - 0.223s success gatsby-plugin-react-i18next: create node: hu-hu/translation - 0.229s success gatsby-plugin-react-i18next: create node: ko-kr/translation - 0.236s success gatsby-plugin-react-i18next: create node: ms-my/translation - 0.241s success gatsby-plugin-react-i18next: create node: pt-br/translation - 0.244s success gatsby-plugin-react-i18next: create node: ro-ro/translation - 0.251s success gatsby-plugin-react-i18next: create node: pl-pl/translation - 0.254s success gatsby-plugin-react-i18next: create node: zh-tw/translation - 0.255s success gatsby-plugin-react-i18next: create node: ru-ru/translation - 0.257s success gatsby-plugin-react-i18next: create node: th-th/translation - 0.255s success gatsby-plugin-react-i18next: create node: zh-my/translation - 0.256s success gatsby-plugin-react-i18next: create node: tr-tr/translation - 0.256s success gatsby-plugin-react-i18next: create node: vn-vn/translation - 0.254s success gatsby-plugin-react-i18next: create node: id-id/translation - 0.253s success Checking for changed pages - 0.001s success source and transform nodes - 212.683s success building schema - 0.846s success Building champion pages - 1.570s success Building champion skin pages - 8.572s success Building league of legends pages - 8.586s success Building Creator redirects - 0.018s success createPages - 10.185s warn Warn: updating default matchPath for all pages /en-us/404/ success createPagesStatefully - 0.136s info Total nodes: 66570, SitePage nodes: 44115 (use --verbose for breakdown) ```

psykzz · 2022-11-10T10:36:10Z

Whats next for this PR?

pieh · 2022-12-09T12:06:48Z

packages/gatsby/src/redux/actions/public.js

-  if (!parent.children.includes(child.id)) {
-    parent.children.push(child.id)
-  }
+  createParentChildLinkBatcher.add({ parent, child })

-  return {
-    type: `ADD_CHILD_NODE_TO_PARENT_NODE`,
-    plugin,
-    payload: parent,
-  }
+  return []


we force flush batcher just after sourceNodes lifecycle and while this is where we expect most of createParentChildLink calls to happen they will not always be just there (for example editing files while dev server is running will randomly update File nodes + run onCreateNode for them outside of sourceNodes). This makes it possible to not create actual link until batch grow large enough to be processed or we sourceNodes is triggered for different reason.

Is it possible to only do batching while during sourceNodes and use previous setup for anything else?

psykzz · 2023-06-29T14:57:12Z

Is there any updates for this PR?

feat(gatsby-transformer-json): Speed up creating nodes for arrays

7a45d61

gatsbot bot added the status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer label Apr 27, 2022

KyleAMathews added 2 commits April 27, 2022 12:04

100 is about 33% faster than 1000

f85f07b

Add type cache

c409697

LekoArts added topic: plugins Related to plugin system, themes & catch-all for plugins that don't have a label and removed status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer labels May 3, 2022

KyleAMathews added 3 commits May 31, 2022 15:56

Merge branch 'master' into faster-transformer-json

a761e9c

Fix creating IDs for nodes

880951e

Debounce writing updated parent node

7ceceea

Invoke on leading and trailing to ensure the parent node's children a…

cbfe597

…rray is updated at least once before schema creation

Fix key for debounce fn

74e12dc

pieh reviewed Jun 1, 2022

View reviewed changes

packages/gatsby/src/datastore/lmdb/lmdb-datastore.ts Outdated Show resolved Hide resolved

pieh reviewed Jun 1, 2022

View reviewed changes

packages/gatsby/src/datastore/lmdb/lmdb-datastore.ts Outdated Show resolved Hide resolved

KyleAMathews added 3 commits June 1, 2022 10:43

batch actions instead of timeout on writing

6fa9892

Keep old behavior for tests

27ff437

Setting batch count of 1 seems to work

81e96b6

KyleAMathews added 2 commits June 13, 2022 13:59

Merge branch 'master' into faster-transformer-json

bc76325

Merge branch 'master' into faster-transformer-json

cc04b88

KyleAMathews changed the title ~~feat(gatsby-transformer-json): Speed up creating nodes for arrays~~ feat(gatsby): Fix regression from v3 -> v4 when creating many child nodes from one parent Jun 29, 2022

Merge branch 'master' into faster-transformer-json

ce78af4

KyleAMathews added the topic: performance Related to runtime & build performance label Jun 29, 2022

Merge branch 'master' into faster-transformer-json

62f56da

Merge branch 'master' into faster-transformer-json

e579e74

LekoArts added 4 commits December 9, 2022 11:22

Merge branch 'master' into faster-transformer-json

80b449f

fix csv transformer

f2dd1ff

remove unnecessary await

f8dcb3c

correct typescript

9485ac0

LekoArts changed the title ~~feat(gatsby): Fix regression from v3 -> v4 when creating many child nodes from one parent~~ feat(gatsby): Optimize creating many child nodes from one parent Dec 9, 2022

pieh reviewed Dec 9, 2022

View reviewed changes

fix csv tests

8e9b755

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(gatsby): Optimize creating many child nodes from one parent #35504

feat(gatsby): Optimize creating many child nodes from one parent #35504

KyleAMathews commented Apr 27, 2022 •

edited by LekoArts

Loading

pogo19 commented May 3, 2022

pogo19 commented May 3, 2022

KyleAMathews commented May 31, 2022 •

edited

Loading

KyleAMathews commented Jun 1, 2022

KyleAMathews commented Jun 1, 2022

pogo19 commented Jun 2, 2022 •

edited

Loading

KyleAMathews commented Jun 3, 2022

pogo19 commented Jul 6, 2022

pogo19 commented Sep 13, 2022

psykzz commented Sep 17, 2022

KyleAMathews commented Oct 12, 2022

psykzz commented Oct 12, 2022

psykzz commented Oct 12, 2022

psykzz commented Nov 10, 2022

pieh Dec 9, 2022 •

edited

Loading

psykzz commented Jun 29, 2023

feat(gatsby): Optimize creating many child nodes from one parent #35504

Are you sure you want to change the base?

feat(gatsby): Optimize creating many child nodes from one parent #35504

Conversation

KyleAMathews commented Apr 27, 2022 • edited by LekoArts Loading

Description

Additional changes

pogo19 commented May 3, 2022

pogo19 commented May 3, 2022

KyleAMathews commented May 31, 2022 • edited Loading

KyleAMathews commented Jun 1, 2022

KyleAMathews commented Jun 1, 2022

pogo19 commented Jun 2, 2022 • edited Loading

KyleAMathews commented Jun 3, 2022

pogo19 commented Jul 6, 2022

pogo19 commented Sep 13, 2022

psykzz commented Sep 17, 2022

KyleAMathews commented Oct 12, 2022

psykzz commented Oct 12, 2022

Setup

Logs

psykzz commented Oct 12, 2022

psykzz commented Nov 10, 2022

pieh Dec 9, 2022 • edited Loading

Choose a reason for hiding this comment

psykzz commented Jun 29, 2023

KyleAMathews commented Apr 27, 2022 •

edited by LekoArts

Loading

KyleAMathews commented May 31, 2022 •

edited

Loading

pogo19 commented Jun 2, 2022 •

edited

Loading

pieh Dec 9, 2022 •

edited

Loading