Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(gatsby-source-filesystem): dont JSON parse/stringify the node #27597

Merged
merged 6 commits into from
Oct 27, 2020

Conversation

pvdz
Copy link
Contributor

@pvdz pvdz commented Oct 22, 2020

This seems to make a few seconds difference at scale. It's also highly unnecessary. And perhaps it has a butterfly effect at scale wrt the GC.

Still benchmarking to confirm the delta. Roughly speaking, the 64k gabe-fs-text seems to save ~2s for this change (which is pretty minor in the grand scheme of things).

@gatsbot gatsbot bot added the status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer label Oct 22, 2020
@pvdz pvdz added topic: performance Related to runtime & build performance topic: scaling builds and removed status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer labels Oct 22, 2020
@pvdz
Copy link
Contributor Author

pvdz commented Oct 22, 2020

gabe-fs-text wins about 1.5s at 64k files with this.

Copy link
Contributor

@vladar vladar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can have a snapshot-based test for a dummy file and directory so that we could see actual before/after structures?

Somewhere here:

Edit: looks like the only existing test in __tests__/create-file-node has no assertions 🤔

@pvdz pvdz requested a review from a team as a code owner October 26, 2020 15:26
@pvdz
Copy link
Contributor Author

pvdz commented Oct 26, 2020

I've added a snapshot test for this. The snapshot was the same on master.

@pvdz pvdz requested a review from vladar October 26, 2020 16:18
@pvdz
Copy link
Contributor Author

pvdz commented Oct 26, 2020

There are some windows pathing problems so I've tried to work around it. Hope this holds. Before state:
image

Copy link
Contributor

@vladar vladar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pvdz You can do this snapshot this way:

expect(node).toMatchInlineSnapshot({
  accessTime: expect.any(String),
  atime: expect.any(String),
  atimeMs: expect.any(Number),
  absolutePath: expect.any(String),
  dir: expect.any(String),
  root: expect.any(String),
})

Then re-run and it will generate the universal snapshot (it will ignore actual values but at least all the keys will be there).

The sad part is that we use jest-serializer-path to normalize paths in snapshots but it doesn't work here because we slash paths in createFileNode and so on windows jest-serializer-path doesn't understand that those are paths 😬

As for dates - they seem to be working in CI but not locally for some reason (in CI dates have a timestamp 0 but locally they fill it with the current date value for me)

@pvdz
Copy link
Contributor Author

pvdz commented Oct 27, 2020

I'm not sure I like that any better than my current solution. I mean I'm also not very happy with having to create a fake file and all that, but the alternative to that is mock out more of fs which I don't want to do either.

At some point a test is too tightly bound to the implementation rather than the feature and it becomes a burden to maintain. I think mocking out fs.stat is a fair tradeof. I could even omit half the props on it since the test (due to its mocks) doesn't really care about that. Just that these Date props are serialized.

Anyways, I think it's fine like this. The test passes.

@pvdz pvdz requested a review from vladar October 27, 2020 07:57
Copy link
Contributor

@vladar vladar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nicely done, thanks! 👍

Copy link
Contributor

@wardpeet wardpeet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@pvdz pvdz merged commit ffbec4c into master Oct 27, 2020
@LekoArts LekoArts deleted the fs-cfn-nojson branch October 27, 2020 10:19
mwfrost pushed a commit to mwfrost/gatsby that referenced this pull request Apr 20, 2023
…atsbyjs#27597)

* perf(gatsby-source-filesystem): dont JSON parse/stringify the node

* Explicit serialization, remove redundant keys (they are splat anyways)

* Explicitly splat stats object

* Add a snapshot test

* Attempt to make the test more generic so it works on Windows too
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic: performance Related to runtime & build performance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants