Non-blocking garbage collector #5149
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
edolstra
force-pushed
the
non-blocking-gc
branch
2 times, most recently
from
02fccb3 to
71d2a14
Compare
|
cc @domenkozar maybe relevant for cachix watch-store? |
|
Will Nix then consider referrers paths a valid? |
|
I hope referrers to a path will still prevent deletions! |
|
Yes, if a path has any refererrs, then deletion will fail. Previously it would recursively delete unreachable referrers (especially dangerous in combination with |
Mic92
reviewed
Aug 25, 2021
Mic92
reviewed
Aug 25, 2021
Mic92
reviewed
Aug 25, 2021
Mic92
reviewed
Aug 25, 2021
roberth
reviewed
Aug 30, 2021
dtzWill
reviewed
Sep 1, 2021
edolstra
force-pushed
the
non-blocking-gc
branch
from
2d68855 to
c8147c1
Compare
edolstra
force-pushed
the
non-blocking-gc
branch
2 times, most recently
from
7a06d24 to
c63af2f
Compare
The garbage collector no longer blocks other processes from adding/building store paths or adding GC roots. To prevent the collector from deleting store paths just added by another process, processes need to connect to the garbage collector via a Unix domain socket to register new temporary roots.
Co-authored-by: Jörg Thalheim <Mic92@users.noreply.github.com>
This test broke the assumption that the hash parts of store paths are unique.
(where "referrers" includes the reverse of derivation outputs and derivers). Now we do a full traversal to look if we can reach any root. If not, all paths reached can be deleted.
edolstra
force-pushed
the
non-blocking-gc
branch
from
75115fc to
35c98a5
Compare
This fixes a bug in the garbage collector where if a path
/nix/store/abcd-foo is valid, but we do a
isValidPath("/nix/store/abcd-foo.lock") first, then a negative entry
for /nix/store/abcd is added to pathInfoCache, so /nix/store/abcd-foo
is subsequently considered invalid and deleted.
The client thread can't just delete its own thread object from connections, it has to detach it.
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/tweag-nix-dev-update-20/15727/1 |
edolstra
force-pushed
the
non-blocking-gc
branch
from
b3837b9 to
33d04e8
Compare
13 tasks
edolstra
added a commit
that referenced
this pull request
Dec 13, 2021
andersk
reviewed
Jun 10, 2022
Comment on lines
+139
to
+146
| } catch (SysError & e) { | ||
| /* The garbage collector may have exited, so we need to | ||
| restart. */ | ||
| if (e.errNo == EPIPE) { | ||
| debug("GC socket disconnected"); | ||
| state->fdRootsSocket.close(); | ||
| goto restart; | ||
| } |
There was a problem hiding this comment.
This is missing a rethrow for other error codes (#6648).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This makes it possible to run builds when the garbage collector is running. So the message
waiting for the big garbage collector lock...is a thing of the past.Processes that start while the GC is running inform the GC process by connecting to the latter via a Unix domain socket (
/nix/var/nix/gc-socket/socket). They write a path to the socket and then read a single-byte response to synchronize. After receiving the response, the GC process knows about the new root and won't delete it.Note:
nix-store --delete <path>will no longer delete the referrers of<path>. That was probably not expected behaviour anyway.