libfetchers/git-utils: Do not refresh pack files in GitFileSystemObje… #14696
+35
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
fetching
xokdvium
force-pushed
the
even-faster-tarball-cache
branch
from
77473ff to
6fd8f4c
Compare
Contributor
Author
|
Impact on my machine (with the on zfs on an nvme ssd, I expect this to be much more pronounced on systems with slower drives):
(Before) (After) |
Radvendii
approved these changes
Dec 2, 2025
This was referenced Dec 2, 2025
…ctSink This leads to incredibly wasteful refreshes (see [^]) when oids are not found. Since we are writing the pack files only once per unpacking we should not bother with this refreshing at all. This brings down the number of syscalls during `nix flake metadata "https://releases.nixos.org/nixos/25.05/nixos-25.05.813095.1c8ba8d3f763/nixexprs.tar.xz" --store "dummy://?read-only=false"` Down from 576334 to just 6235 (100x less syscalls): (Before) % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ------------------ 32.98 0.625288 3 162898 getdents64 29.58 0.560686 3 163514 81917 openat 15.01 0.284509 3 81819 186 newfstatat 10.99 0.208349 2 81601 close 10.56 0.200145 2 81552 fstat All these are coming from [2] and are totally useless. (After) % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ------------------ 76.47 0.108558 247 438 20 futex 6.55 0.009292 18 513 munmap 3.30 0.004680 7 639 492 openat 2.68 0.003803 10 359 write 2.30 0.003268 2 1146 read 2.26 0.003215 3 870 mmap [^]: https://github.com/libgit2/libgit2/blob/58d9363f02f1fa39e46d49b604f27008e75b72f2/include/git2/sys/odb_backend.h#L68-L75 [2]: https://github.com/libgit2/libgit2/blob/58d9363f02f1fa39e46d49b604f27008e75b72f2/src/libgit2/odb_pack.c#L517-L546
xokdvium
force-pushed
the
even-faster-tarball-cache
branch
from
6fd8f4c to
d1f9fe9
Compare
Ericson2314
approved these changes
Dec 3, 2025
Radvendii
reviewed
Dec 3, 2025
| unset it in the vtable. libgit2 does nothing if it's a nullptr: | ||
| https://github.com/libgit2/libgit2/blob/58d9363f02f1fa39e46d49b604f27008e75b72f2/src/libgit2/odb.c#L1922 | ||
| */ | ||
| packfileOdbRefresh = std::exchange(backend->refresh, nullptr); |
Contributor
There was a problem hiding this comment.
Is this different than
packfileOdbRefresh = backend->refresh;
backend->refresh = nullptr;
?
Comment on lines
+1343
to
+1344
| /* We are done writing blobs, can restore refresh functionality. */ | ||
| backend->refresh = packfileOdbRefresh; |
Contributor
There was a problem hiding this comment.
I assume that the status of repo->packBackend never changes for a given repo?
edolstra
approved these changes
Dec 3, 2025
Member
|
I don't know if it's this or one of the other git optimizations, but I see errors about missing commits from repos. Unfortunately I didn't copy the exact error and now it went away. will do the next time i see it. |
Member
|
Unclear now if this was the actual root cause or it's an issue maybe fixed by #14620 Will do more testing after I rebased again. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…ctSink
Motivation
This leads to incredibly wasteful refreshes (see ^) when oids are not found. Since we are writing the pack files only once per unpacking we should not bother with this refreshing at all.
This brings down the number of syscalls during
nix flake metadata "https://releases.nixos.org/nixos/25.05/nixos-25.05.813095.1c8ba8d3f763/nixexprs.tar.xz" --store "dummy://?read-only=false"Down from 576334 to just 6235 (100x less syscalls):
(Before)
All these are coming from 2 and are totally useless.
(After)
Context
Natural follow-up to #14689.
Add 👍 to pull requests you find important.
The Nix maintainer team uses a GitHub project board to schedule and track reviews.