Lazy trees #6530

edolstra · 2022-05-12T19:10:40Z

Features

Flake inputs are now only copied to the Nix store if that's actually needed (fixing Copy local flakes to the store lazily #3121). That is, an operation like nix build nixpkgs#hello no longer copies the nixpkgs flake to the Nix store. Only an expression like src = ./. or nix.registry.nixpkgs.flake = nixpkgs; will cause an entire flake to be copied.
fetchTree can now apply patches to a tree, e.g. fetchTree { ...; patches = [ ./foo.patch ./bar.patch ]; } (Support flake references to patches #3920). Patches are applied in memory, so this doesn't require the entire tree to be materialized to disk.
The github fetcher now fetches zip archives instead of tarballs, and does not unpack them to disk. Depending on the filesystem, this can save a lot of disk space. (E.g. on ext4 with a 4 KiB block size, an unpacked Nixpkgs tree takes 252 MiB, whereas the zip archive takes 43 MiB.

Design

The evaluator no longer accesses the filesystem directly. Instead, it uses a virtual filesystem abstraction called InputAccessor, which has FS operations like readFile() . You can get the InputAccessor for an Input by calling Input::lazyFetch(), which unlike Input::fetch() does not (necessarily) copy the input to the Nix store.

The following InputAccessors currently exist:

FSInputAccessor: This directly accesses the host filesystem, optionally rooted at some prefix. This is used for both non-flake evaluation (e.g. nix-build ./foo.nix), for path:// flakes, and for (dirty or clean) Git working directories (git+file://). It optionally takes a set of allowed paths, which is used by the Git input type to ensure that you can't access untracked files.
MemoryInputAccessor: An in-memory filesystem, used to provide <nix/fetchurl.nix>.
ZipInputAccessor: An accessor that reads directly from a zipfile, using libzip. Used by the github fetcher.
PatchingInputAccessor: An adaptor that wraps another accessor whose readFile() applies a patch to the file returned by the underlying accessor.

Paths are represented inside the evaluator using the SourcePath type, which is a tuple of an InputAccessor and a path relative to the root of the input. So the SourcePath {input->lazyFetch(), "/flake.nix"} represents the file flake.nix inside some input.

Incompatible changes

The outPath attribute returned by fetchTree et al. is no longer a store path but a path (i.e. a SourcePath). This can cause some breakage, e.g. in NixOS, setting

nix.registry.nixpkgs.flake = nixpkgs;

causes the error A definition for option 'nix.registry.nixpkgs.to.path' is not of type 'string or signed integer or boolean or package'. Instead you have to write:

nix.registry.nixpkgs.flake = "${nixpkgs}";

to force the argument to be a store path.

To do

Fix many test failures.
Fix showing source code in error messages. Currently the error formatter doesn't know anything about SourcePaths.
Give a helpful message when trying to use untracked files in a git repo (Be smart and helpful when some files are missing during the evaluation #4507).
Fix the evaluation cache.
Fix flake locking.
Get rid of the narHash attribute in flake.lock? Computing it is expensive since it requires reading the entire source tree. However, we need it to be able to substitute flake inputs, but maybe we don't care about that.
What to do with path:// inputs? They're only locked by narHash, so if we don't have narHash, we can't lock path:// inputs.
Fix/improve subflake handling.
Update release notes / docs.

Context

Closes Copy local flakes to the store lazily #3121
Closes Support flake references to patches #3920
Closes Restrict builtins.isStorePath #5973

jaen · 2022-05-13T15:11:52Z

Sorry if this is dumb question, but does it have any bearing on support for git LFS and similar features (git-crypt git annex)? I could imagine the virtual file system accessor transparently handling reading the actual large file instead of the pointer file (this is what currently happens for me when I try to use git LFS for my secrets with sops-nix and it breaks the rebuild) to make this work.

Could this be a right use-case for this feature, or am I misunderstanding it?

blaggacao · 2022-05-18T22:10:44Z

What to do with path:// inputs? They're only locked by narHash, so if we don't have narHash, we can't lock path:// inputs. maybe we don't care about that.

I agree. We don't care. This greately improves the UX of relative flakes within a repository.

Absolut paths are probably only sensible as a development helper.

roberth · 2022-05-26T09:43:10Z

Absolut paths are probably only sensible as a development helper.

and the relative paths like subflakes can be locked by locking their parent flake instead.

nixos-discourse · 2022-06-03T09:48:49Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-31/19481/1

layus · 2022-06-03T19:00:25Z

src/libfetchers/zip-input-accessor.cc

+    std::string readFile(const CanonPath & path) override
+    {
+        if (lstat(path).type != tRegular)
+            throw Error("file '%s' is not a regular file");


nit:

-throw Error("file '%s' is not a regular file"); +throw Error("file '%s' is not a regular file", path);

I am getting this error while running update:

nix run github:edolstra/nix?ref=lazy-trees#nix -- flake update --show-trace warning: Git tree '/my/project/dir' is dirty error: file '%s' is not a regular file … while updating the flake input 'mach-nix' … while updating the lock file of flake 'git+file:///my/project/dir?ref=refs%2fheads%2fmy-current-branch'

and mach-nix input is defined like this:

mach-nix.url = "github:DavHau/mach-nix"; mach-nix.inputs.nixpkgs.follows = "nixpkgs"; mach-nix.inputs.flake-utils.follows = "flake-utils";

Very strange, the error is with flake.lock:

error: file '/flake.lock' is not a regular file … while updating the flake input 'mach-nix' … while updating the lock file of flake 'git+file:///my/project/dir?ref=refs%2fheads%2fmy-current-branch'

Oh, mach-nix has a symlink for the flake.lock. This explains the error.

https://github.com/DavHau/mach-nix/blob/master/flake.lock

I guess readFile has to resolve symlinks now. This in turns asks the question of 1) do we want to support that, 2) Should we prevent symlinks that go out of the flake boundary and 3) do we accept that lazy flakes (that live in a zip archive) will have a different behavior regarding symlinks that escape the flake than flakes that live plainly in the store.

@edolstra This patch works for me 🥳 : layus@44d5bd4

I think it is safe because paths are rooted at the archive root. But is it always correct ? Could be.

nrdxp · 2022-06-03T23:10:57Z

These changes seem to disallow a flake from setting a different version of itself as an input via a url. This was allowed before. To reproduce just run nix flake lock github:input-output-hk/cardano-node as an example. The following inputs triggers the problem:
https://github.com/input-output-hk/cardano-node/blob/7ee6467e288aee5021f048a1c898c05881060e4b/flake.nix#L44-L49

Not sure how or why this worked before and not now, just kinda an odd find I thought was worth mentioning.

nixos-discourse · 2022-06-20T13:21:00Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-32/19865/1

icetan · 2022-06-27T13:24:47Z

Wanted to try out relative paths to be able to use subflakes in a practical way, but I ran into some problems.

Root flake file flake.nix:

{
  inputs = {
    utils.url = "github:numtide/flake-utils/bee6a7250dd1b01844a2de7e02e4df7d8a0a206c";
    sub1.url = "path:./sub1";
  };
  outputs = { nixpkgs, utils, sub1, ... }:
    utils.lib.eachDefaultSystem (system: {
      packages = {
        inherit (sub1.packages.${system}) hello;
      };
    });
}

Subflake file sub1/flake.nix:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/9a17f325397d137ac4d219ecbd5c7f15154422f4";
    utils.url = "github:numtide/flake-utils/bee6a7250dd1b01844a2de7e02e4df7d8a0a206c";
  };
  outputs = { nixpkgs, utils, ... }:
    utils.lib.eachDefaultSystem (system:
      let
        legacyPackages = import nixpkgs { inherit system; };
      in
      {
        packages = {
          inherit (legacyPackages) hello;
        };
      });
}

When I run nix flake show with latest from lazy-trees branch:

nix run github:edolstra/nix/04cb555aebfff68732cc7f0120def20449515eea -- flake show --show-trace

I get the output:

path:/home/icetan/src/nix/test-subflakes
└───packages
    ├───aarch64-darwin
error: in pure evaluation mode, 'fetchTree' requires a locked input, at (string):16:18

       … while realising the context of a path

       at «string»:20:19:

           19|
           20|           flake = import (sourceInfo + (if subdir != "" then "/" else "") + subdir + "/flake.nix");
             |                   ^
           21|

       … while evaluating anonymous lambda

       at «string»:10:13:

            9|     builtins.mapAttrs
           10|       (key: node:
             |             ^
           11|         let

       … from call site

       … while evaluating anonymous lambda

       at «string»:23:25:

           22|           inputs = builtins.mapAttrs
           23|             (inputName: inputSpec: allNodes.${resolveInput inputSpec})
             |                         ^
           24|             (node.inputs or {});

       … from call site

I expect something more like when running nix flake show using nix (Nix) 2.9.1:

path:/home/icetan/src/nix/test-subflakes?lastModified=1656334768&narHash=sha256-ODEqelm5y%2fVQkVkX11EWMWOHFpH+8wMZpEySuE0lPWY=
└───packages
    ├───aarch64-darwin
    │   └───hello: package 'hello-2.12'
    ├───aarch64-linux
    │   └───hello: package 'hello-2.12'
    ├───i686-linux
    │   └───hello: package 'hello-2.12'
    ├───x86_64-darwin
    │   └───hello: package 'hello-2.12'
    └───x86_64-linux
        └───hello: package 'hello-2.12'

edolstra · 2022-06-27T16:43:38Z

@icetan Yeah, subflakes are broken, I'll fix it.

nixos-discourse · 2022-08-03T23:08:41Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/running-nixos-rebuild-does-not-add-new-files-have-i-misunderstood-its-purpose/20664/10

nixos-discourse · 2022-08-15T20:51:59Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/import-custom-directories-files-to-package-path-on-nix-store/20991/7

lheckemann · 2022-08-17T13:34:27Z

src/libcmd/command.cc

 {
+    auto path = file.getPhysicalPath();
+    if (!path)
+        throw Error("cannot open '%s' in an editor because it has no physical path", file);


It would be nice to write it to a temporary file to allow at least read-only operations on the file. By no means a blocker, but maybe a FIXME would be appropriate here?

src/libcmd/command.cc

lheckemann · 2022-08-17T14:09:39Z

src/libcmd/repl.cc

+                if (auto path = std::get_if<SourcePath>(&pos.origin))
+                    return {*path, pos.line};
+                else
+                    throw Error("'%s' cannot be shown in an editor", pos);


This would happen when the position of the lambda can't be determined, right? I think this error message doesn't reflect that very well.

src/libexpr/eval-cache.cc

lheckemann · 2022-08-18T20:34:03Z

{
  inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-22.05";
  outputs = {nixpkgs, ...}: {
    defaultPackage.x86_64-linux = nixpkgs.legacyPackages.x86_64-linux.steam;
  };
}

Fails to build because of unfree as expected; however, the source location information in the error message is weird (should probably reference the input name if possible?):

error: Package ‘steam’ in /virtual/4/pkgs/games/steam/steam.nix:43 has an unfree license (‘unfreeRedistributable’), refusing to evaluate.

lheckemann · 2022-08-19T09:10:51Z

One unfortunate consequence of this is that you can no longer just open a path mentioned in an error trace, which can make digging into code for debugging errors a lot more painful. Using --override-input to replace e.g. nixpkgs with a local path can probably help with this, but ideally there would be a way to do this for any input type as it was before lazy-trees.

I can think of several fancy ways to work around this, but they all seem quite expensive and this PR is already a lot bigger than it ideally would be; an option to restore the old behaviour of fetching and unpacking inputs to the store would be extremely helpful here.

Fancy too-big-for-this-PR ideas

nix edit could be expanded to allow opening files within inputs read-only
- For extra fanciness, this could be read-write and Nix could output a patch to apply to the input once the editor is done
I previously mentioned this as a joke, but a FUSE filesystem (nix flake mount?) would provide a much more comfortable input-inspection facility and could also provide the ability to generate patches

lheckemann · 2022-08-18T09:20:02Z

src/libexpr/eval.cc

+            evalSettings.restrictEval || evalSettings.pureEval
+            ? std::optional<std::set<CanonPath>>(std::set<CanonPath>())
+            : std::nullopt,


This reads like 3 args for makeFSInputAccessor. It's still not perfect, but maybe

Suggested change

evalSettings.restrictEval || evalSettings.pureEval

? std::optional<std::set<CanonPath>>(std::set<CanonPath>())

: std::nullopt,

evalSettings.restrictEval || evalSettings.pureEval

? std::optional<std::set<CanonPath>>(std::set<CanonPath>())

: std::nullopt,

What would be nicer is factoring it out into an allowedPaths variable but I'm not sure we can do that in this constructor context (maybe rootFS should just be initialised in the constructor's body?)

lheckemann · 2022-08-19T11:00:24Z

src/libexpr/eval.cc

@@ -525,14 +532,12 @@ EvalState::~EvalState()

 void EvalState::allowPath(const Path & path)
 {
-    if (allowedPaths)
-        allowedPaths->insert(path);
+    rootFS->allowPath(CanonPath(path)); // FIXME


Is this the symlinks-and-security-relevant stuff you mentioned?

lheckemann · 2022-08-19T14:03:18Z

src/libexpr/eval.cc

@@ -2508,7 +2477,8 @@ void EvalState::printStats()
                    else
                        obj.attr("name", nullptr);
                    if (auto pos = positions[fun->pos]) {
-                        obj.attr("file", (std::string_view) pos.file);
+                        if (auto path = std::get_if<SourcePath>(&pos.origin))


I think it would be nice to cover the other variants (or at least stdin and string) in some way here too -- maybe even add some further metadata to allow distinguishing --apply from --expr strings. Nothing that needs to be resolved in this PR though IMHO.

src/libexpr/nixexpr.hh

src/libexpr/eval.hh

nixos-discourse · 2022-08-20T18:25:10Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/proper-way-of-applying-patch-to-system-managed-via-flake/21073/24

lheckemann · 2022-08-21T10:17:22Z

Not sure if intentional or not, but this fixes referring to local bare repos like nix flake metadata git+file:///path/to/repo.git 👍

Atry · 2022-08-24T02:45:39Z

Would nix develop copy the entire source if the devShell has an inputsFrom attribute from a derivation whose src is ./.?

For example:

Atry · 2022-08-24T02:49:24Z

I think it's OK to copy the entire source tree for a clean build, because it is supposed to be slow when it's huge, but it's better not to copy the tree for a development shell

roberth · 2022-08-24T08:31:29Z

but it's better not to copy the tree for a development shell

The hhvm package "looks" like it requires the copy, but the expression should be lazy enough to avoid evaluating src.
A similar expression does not evaluate .src.

$ nix-instantiate --expr 'with import <nixpkgs> {}; mkShell { inputsFrom = [ (stdenv.mkDerivation { name = "hi"; src = throw "dont"; }) ]; }' --show-trace
/nix/store/n3ssq0mx06iiw2bs59mhnj8am1v1zfya-nix-shell.drv

erikarvstedt · 2024-07-29T12:14:08Z

This fails when NIX_PATH contains flake references:

NIX_PATH=nixpkgs=flake:nixpkgs nix eval --impure --expr '
(import <nixpkgs/nixos> {
  configuration = {};
}).vm.drvPath
'

Result:

error: cannot decode virtual path '/nix/store/lazylazy000000000000000000000004-source/pkgs/development/compilers/llvm/17/llvm/gnu-install-dirs.patch'

With this, we have essentially achieved NixOS/nix#6530 in pure nix. The only remaining "impurity" in this sense would be the path passed to the composer function, but this is actually a good thing, since it doesn't eagerly copy everything to the nix store like flakes, but instead only makes pure copies of files when they are actually accessed. If the path doesn't exist on the file-system we will immediately fail anyway, so it's "impurity" is irrelevant. There will be no way for the user to escape the module system with these new enforcements, so that means that we basically have a totally pure nix evaluation without the high cost of having to copy the entire repo into the /nix/store eagerly.

nixos-discourse · 2024-09-24T21:29:27Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-are-flake-dependencies-fetched/52638/4

nixos-discourse · 2024-10-27T17:39:41Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-team-member-suggests-removing-flakes-data-suggest-it-isnt-a-good-idea-please-remove-the-experimental-flag-instead/54959/78

nixos-discourse · 2024-10-28T20:24:58Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-team-member-suggests-removing-flakes-data-suggest-it-isnt-a-good-idea-please-remove-the-experimental-flag-instead/54959/104

waiting for NixOS/nix#6530 Signed-off-by: saygo-png <saygo.mail@proton.me>

This comment was marked as outdated.

Sign in to view

edolstra mentioned this pull request May 27, 2022

Fix a segfault in the git fetcher #6582

Merged

Atemu mentioned this pull request Jun 3, 2022

Copy local flakes to the store lazily #3121

Open

layus reviewed Jun 3, 2022

View reviewed changes

This was referenced Jul 5, 2022

Improved flake support hercules-ci/arion#129

Open

nixopsUnstable broke after upgrade recently NixOS/nixpkgs#139532

Closed

roberth mentioned this pull request Jul 18, 2022

New builtin: pathType #3096

Closed

This was referenced Aug 2, 2022

[flake] nix flake init failed with ~/.local/share/nix/root used as a chroot store #6853

Open

Don't ignore unstaged files in local flakes #6858

Open

lheckemann reviewed Aug 17, 2022

View reviewed changes

This comment was marked as resolved.

Sign in to view

This comment was marked as outdated.

Sign in to view

lheckemann reviewed Aug 19, 2022

View reviewed changes

roberth mentioned this pull request Jun 28, 2024

Multithreaded evaluator #10938

Draft

edolstra and others added 4 commits July 2, 2024 16:56

Merge remote-tracking branch 'origin/master' into lazy-trees

57b2cc2

Formatting

1b20945

Busywork

53fabdc

Merge branch 'master' into lazy-trees

34e4a61

roberth mentioned this pull request Jul 5, 2024

Source roots result in /nix/store/<hash>-<hash>-source #10627

Open

Merge remote-tracking branch 'origin/master' into lazy-trees

0d3a157

emilazy mentioned this pull request Jul 14, 2024

Cargo.lock considered harmful NixOS/nixpkgs#327063

Closed

This was referenced Jul 23, 2024

fetchTree/github:: add merge argument #11160

Open

Lazy, async instantiation and fetching (tracking) #11161

Open

roberth mentioned this pull request Jul 31, 2024

File '/dev/full' has an unsupported type in / in Docker #11217

Open

roberth mentioned this pull request Aug 10, 2024

Update lookup-path.md #11276

Open

roberth mentioned this pull request Aug 25, 2024

libgit2, GitRepo: Write (thin) packfiles #11330

Merged

4 tasks

This was referenced Sep 7, 2024

Fetch Git submodules lazily #11425

Draft

fetcher cache sqlite error: table Cache has no column named domain #10985

Closed

domenkozar mentioned this pull request Oct 7, 2024

Installing external Flakes partly broken since 1.1/ devenv-2.24 cachix/devenv#1456

Open

Artturin mentioned this pull request Oct 16, 2024

Fix warning on nix lazy-trees branch astro/deadnix#88

Merged

Frontear mentioned this pull request Oct 19, 2024

Fix delay with unfree package set llakala/nixos#68

Open

saygo-png added a commit to saygo-png/nixos that referenced this pull request Nov 7, 2024

refactor: No more devenv shells

563ecbb

waiting for NixOS/nix#6530 Signed-off-by: saygo-png <saygo.mail@proton.me>

edolstra added 6 commits November 18, 2024 14:58

Merge remote-tracking branch 'origin/master' into lazy-trees

800bc9c

Remove unused file

9ed105a

Merge remote-tracking branch 'origin/master' into lazy-trees

2ab5278

Merge remote-tracking branch 'origin/master' into lazy-trees

879f6b6

Reduce diff

355cb56

Merge remote-tracking branch 'origin/master' into lazy-trees

250813a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lazy trees #6530

Lazy trees #6530

edolstra commented May 12, 2022 •

edited by roberth

Loading

jaen commented May 13, 2022

This comment was marked as outdated.

blaggacao commented May 18, 2022

roberth commented May 26, 2022

nixos-discourse commented Jun 3, 2022

layus Jun 3, 2022

layus Jun 9, 2022

layus Jun 9, 2022

layus Jun 9, 2022

layus Jun 9, 2022

nrdxp commented Jun 3, 2022 •

edited

Loading

nixos-discourse commented Jun 20, 2022

icetan commented Jun 27, 2022

edolstra commented Jun 27, 2022

nixos-discourse commented Aug 3, 2022

nixos-discourse commented Aug 15, 2022

lheckemann Aug 17, 2022

lheckemann Aug 17, 2022

This comment was marked as resolved.

lheckemann commented Aug 18, 2022

lheckemann commented Aug 19, 2022

This comment was marked as outdated.

lheckemann Aug 18, 2022

lheckemann Aug 19, 2022

lheckemann Aug 19, 2022

nixos-discourse commented Aug 20, 2022

lheckemann commented Aug 21, 2022

Atry commented Aug 24, 2022

Atry commented Aug 24, 2022

roberth commented Aug 24, 2022

erikarvstedt commented Jul 29, 2024

nixos-discourse commented Sep 24, 2024

nixos-discourse commented Oct 27, 2024

nixos-discourse commented Oct 28, 2024

Lazy trees #6530

Are you sure you want to change the base?

Lazy trees #6530

Conversation

edolstra commented May 12, 2022 • edited by roberth Loading

Features

Design

Incompatible changes

To do

Context

jaen commented May 13, 2022

This comment was marked as outdated.

blaggacao commented May 18, 2022

roberth commented May 26, 2022

nixos-discourse commented Jun 3, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nrdxp commented Jun 3, 2022 • edited Loading

nixos-discourse commented Jun 20, 2022

icetan commented Jun 27, 2022

edolstra commented Jun 27, 2022

nixos-discourse commented Aug 3, 2022

nixos-discourse commented Aug 15, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as resolved.

lheckemann commented Aug 18, 2022

lheckemann commented Aug 19, 2022

This comment was marked as outdated.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nixos-discourse commented Aug 20, 2022

lheckemann commented Aug 21, 2022

Atry commented Aug 24, 2022

Atry commented Aug 24, 2022

roberth commented Aug 24, 2022

erikarvstedt commented Jul 29, 2024

nixos-discourse commented Sep 24, 2024

nixos-discourse commented Oct 27, 2024

nixos-discourse commented Oct 28, 2024

edolstra commented May 12, 2022 •

edited by roberth

Loading

nrdxp commented Jun 3, 2022 •

edited

Loading