Make nix copy parallel again #6612
Conversation
Also the code is a bit of a mess right now |
Bring back the possibility to copy CA paths with no reference (like the outputs of FO derivations or stuff imported at eval time) between stores that have a different prefix.
|
Thank you so much for doing this @thufschmitt! This change has been invaluable in helping me quickly copy things to a binary cache I set up for CI/CD. |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/tweag-nix-dev-update-32/19865/1 |
Like the old implem did (and like you'd want it to be anyways)
This hang for some reason didn't trigger in the Nix build, but did running 'make installcheck' interactively. What happened: * Store::addMultipleToStore() calls a SinkToSource object to copy a path, which in turn calls LegacySSHStore::narFromPath(), which acquires a connection. * The SinkToSource object is not destroyed after the last bytes has been read, so the coroutine's stack is still alive and its destructors are not run. So the connection is not released. * Then when the next path is copied, because max-connections = 1, LegacySSHStore::narFromPath() hangs forever waiting for a connection to be released. The fix is to make sure that the source object is destroyed when we're done with it.
|
Fixed a subtle hang in f0358ed. |
|
Interesting bug! So this is something that |
|
The problem is that |
|
Oh thanks I see now. I think it might be better to move the unique pointer out of the map (leaving behind a null one, I suppose). And then we get the right destructor call by default. |
|
@Ericson2314 You're right, in fact moving it was necessary to get it to build on macOS. |
Make nix copy parallel again
nix copyused to do stuff in parallel. This got removed in #5048 (to allow batching the copies in the ssh store case), but both can actually be kept in parallel with a little effort.This is currently incomplete as it
Disables the batch ssh copyRemoves the ability to copy CA paths from one store prefix to another (I'm curious what the use-case for this is)cc @ConnorBaker