File substitutes #52
File substitutes #52
Conversation
…to get one meta attribute
…t to get the description
…yError when forceValue'd
…ors must be assumed not to be prebuilt
…h, and meta attributes
None of the cases that turned handling off didn't have readOnly set anyway.
…tion in read-only mode, we shouldn't throw an exception just because we're trying to check for substitutable paths
…ith imports from derivations
Rather than relying on magic characters and string scanning to extract information out of a string context, a string's context is now a structured data type. To avoid fragmentation and extra allocations, the context struct and the memory for the strings it points to are all allocated in one call.
Conflicts: src/libexpr/primops.cc
…faces in libstore, and use them to see if a file to be imported has a substitute
…it exists and is needed
…pied-to-store path when necessary
…re actually needed
|
Giving up on this getting merged |
|
Pity to see this closed, but has your thinking re: the problem changed at all since this solution? I find myself wanting the behavior. Would you still implement it this way? |
|
@copumpkin I think as long as we care about read-only evaluation, this kind of approach is the only way to handle "building at eval time". Since doing this work, though, I've come to appreciate the "evaluate at build time" (i.e. recursive nix) approach to the kinds of problems this would solve, especially given the "derivation replacement" model I came up with in my implementation. |
|
@copumpkin This is actually the latest version of the work in #31, sorry |
|
Ah oaky, thanks! @edolstra can we reopen this one then? 😄 |
|
|
||
| aDotNix = builtins.toFile "a.nix" aDotNixString; | ||
|
|
||
| imported = import bar.outPath; |
There was a problem hiding this comment.
why do you need .outPath here?
|
@shlevy when you have some time, do you think you could:
Thanks for all the great work! I hope to be able to revive it and get more of it merged |
|
Reopened so @copumpkin can take over. |
|
So there are multiple things happening in this branch. The first half makes nix-env -q lazier and aware of readonly errors, so if it can complete its query without importing from derivation it will and if it can't it will simply skip that drv as if it threw an assert error. The second half adds the concept of a file substituter, which can provide various bits of information that would otherwise require building during evaluation to determine (though no actual file substituter is added, just the protocol). |
|
@copumpkin Are you planning on reviving this? |
|
@copumpkin will reopen if so |
IPFS Header for ipfs-develop
This branch (which includes my structured-context work and the better errors when importing from derivations work) adds some new substituter operations (--query haveFile, --query fileInfo, and --read) that should provide enough information to determine if a substitute is available for a derivation that requires importing from another derivation to be realised. Essentially, a substituter can serve up information about some files (type, length, recursive hash, etc., used by coerceToString with copyToStore=true) and the full contents of some regular files (used by prim_import), allowing the import from derivation to be done without the full derivation being realised.
This does not update any real substituters to support these new operations, though tests/file-substituter.sh shows the interface they must follow.
A sketch of how this might be used: I import build.nix from nix's tarball, and the substituter gives back the full nix expression which gets parsed and evaled. build.nix builds individual .c files, so the substituter tells me the recursive hash of those files (but it doesn't need the full contents) so nix can determine the hash of compileC derivations.
There are other operations (e.g. readFile) which might want to use this file substituter mechanism in the import-from-derivation case, but import and coerceToString seemed the most important.