-
Notifications
You must be signed in to change notification settings - Fork 10.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
perf(gatsby-source-filesystem): dont JSON parse/stringify the node #27597
Conversation
gabe-fs-text wins about 1.5s at 64k files with this. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we can have a snapshot-based test for a dummy file and directory so that we could see actual before/after structures?
Somewhere here:
describe(`create-file-node`, () => { |
Edit: looks like the only existing test in __tests__/create-file-node
has no assertions 🤔
I've added a snapshot test for this. The snapshot was the same on master. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@pvdz You can do this snapshot this way:
expect(node).toMatchInlineSnapshot({
accessTime: expect.any(String),
atime: expect.any(String),
atimeMs: expect.any(Number),
absolutePath: expect.any(String),
dir: expect.any(String),
root: expect.any(String),
})
Then re-run and it will generate the universal snapshot (it will ignore actual values but at least all the keys will be there).
The sad part is that we use jest-serializer-path
to normalize paths in snapshots but it doesn't work here because we slash
paths in createFileNode
and so on windows jest-serializer-path
doesn't understand that those are paths 😬
As for dates - they seem to be working in CI but not locally for some reason (in CI dates have a timestamp 0 but locally they fill it with the current date value for me)
I'm not sure I like that any better than my current solution. I mean I'm also not very happy with having to create a fake file and all that, but the alternative to that is mock out more of fs which I don't want to do either. At some point a test is too tightly bound to the implementation rather than the feature and it becomes a burden to maintain. I think mocking out Anyways, I think it's fine like this. The test passes. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nicely done, thanks! 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
…atsbyjs#27597) * perf(gatsby-source-filesystem): dont JSON parse/stringify the node * Explicit serialization, remove redundant keys (they are splat anyways) * Explicitly splat stats object * Add a snapshot test * Attempt to make the test more generic so it works on Windows too
This seems to make a few seconds difference at scale. It's also highly unnecessary. And perhaps it has a butterfly effect at scale wrt the GC.
Still benchmarking to confirm the delta. Roughly speaking, the 64k gabe-fs-text seems to save ~2s for this change (which is pretty minor in the grand scheme of things).