fix(gatsby): limit node manifest creation limit

[veryspry]

Description

Context

Node Manifest files are used to tie a unique revision state of a piece of content in a data source to a gatsby page generated from that exact data. In some cases, this is causing upwards of 150K plus node manifest files to get written to disk which can cause performance degradation in certain cases.

How is it done?

This helps PR introduces a limit to the number of node manifest files written to disk during a gatsby build which can be controlled with the env var NODE_MANIFEST_FILE_LIMIT.

If the number of node manifests that are queued to get written to disk is greater than NODE_MANIFEST_FILE_LIMIT, then the node manifest "candidates" are sorted by their updatedAtUTC field timestamp. That sorted list of node manifests is used to queue up file writes file writes are then stopped as soon as the NODE_MANIFEST_FILE_LIMIT has been reached

⚠️ Notes

• The current limit is set to default at 10,000 which is still quite high. We might want to tweak this before merging. Feel free to discuss 🙂
• The integration tests for this were acting a little funny. I still need to spend a little time making sure everything is okay there.

Node Manifest Sorting

I spent some time testing some different sorting approaches and it turns out that the native array sort is almost certainly adequate.

Sorting 1,000,000 node manifests took ~11156 ms while sorting 100,000 took ~914 ms. These numbers seem well within reason in context of a Gatsby build.

[wardpeet]

I don't have enough context but I fear that this might break in subtle ways or I might not understand the feature well enough

[TylerBarnes]

Looking good! 🙌 Just a couple nits

[DanielSLew]

Looks good! This is out of the scope of this PR but maybe we need a way to clear out old manifests as well? As it is now we only clear them on cache clears. But maybe there's more we could be doing to clear them so they don't keep accumulating?

[TylerBarnes]

@DanielSLew it's a good idea but I think out of scope for this PR. We have the manifest ID as part of each page so perhaps when a page is deleted we could delete the manifest 🤔

[veryspry]

Just updated the tests! I realized that they would fail if running all of the integration tests for this (I was only running the one new test while adding new tests and didn't check to make sure that everything would still pass if the whole test file got ran)

[veryspry]

I published a canary with these changes:

gatsby@4.13.0-alpha-node-manifest-limit.15