Lazy attribute names

[infinisil]

Nix could support evaluating attribute sets more lazily, such that e.g. the following works:

(throw "" // { x = 0; }).x
-> 0

Currently this doesn't work because attribute sets are evaluated strictly in all names, meaning all // operations and co. will have to be applied to figure out all names, making the above code throw in the current version.

Making attribute names lazy in that way has the potential to solve many existing problems:

• .extend attributes of package scopes won't necessarily have to evaluate all package attributes. Similarly .override, etc. of packages won't have to call the derivation primop multiple times anymore
• types.lazyAttrsOf might not be needed anymore
• In overlays you'll be able to use self.lib for the attribute definitions if lib is added in a later overlay
• lib.mkIf might not be necessary for the module system anymore
• Potentially more

However this also changes semantics, so this should be thought out well.

I have implemented an initial prototype that supports above evaluation in infinisil@afeda4f. I'll continue with this if I have time.

Ping @edolstra @Profpatsch

---

NOTE (@roberth): To manage expectations, querying a non-existing attribute fundamentally requires evaluation of all attribute names.

• e.g. if attrs?attr then e_lazy else e_strict has to strictly evaluate the attrnames for e_strict.
• { foo, ...}: -style functions have to strictly evaluate all the attrnames in order to preserve their existing semantics.

[infinisil]

Gave this another implementation try in infinisil@5dbf67c, which works much better than the previous one. For example:

let
  lib = import ./lib;

  set1 = lib.makeExtensible (self: builtins.trace "set1" {
    foo = 1;
  });

  set2 = set1.extend (self: super: builtins.trace "set 2" {
    bar = 2;
  });

  set3 = set2.extend (self: super: builtins.trace "set  3" {
    baz = 3;
  });

  set4 = set3.extend (self: super: builtins.trace "set   4" {
    qux = 4;
  });
in set4

Evaluating this in a current Nix version would print

trace: set1
trace: set1
trace: set 2
trace: set1
trace: set 2
trace: set  3
trace: set1
trace: set 2
trace: set  3
trace: set   4
{ __unfix__ = <LAMBDA>; bar = 2; baz = 3; extend = <CODE>; foo = 1; qux = 4; }

Whereas with above commit it just prints:

trace: set1
trace: set 2
trace: set  3
trace: set   4
{ __unfix__ = <LAMBDA>; bar = 2; baz = 3; extend = <CODE>; foo = 1; qux = 4; }

Therefore not evaluating all the overlays multiple times! Unfortunately there is a bug with this implementation which causes it to go into an infinite loop with nixpkgs derivations. Also, this implementation copy-pasted a bunch of code around, and probably leaks some memory and does additional unnecessary allocations here and there. Definitely not ready, but it's getting there :)

[infinisil]

Turns out there was no infinite recursion in the last version, but instead it was so slow that I didn't think it would ever finish!

I worked more on this, with the latest version being in https://github.com/Infinisil/nix/tree/lazy-attr-names-v3. It's still about 40% slower than stable (tested on a firefox instantiation), but it's usable now! There are some other things to do though. I still have high hopes that it can improve the speed of Nix in general, though it might be pretty hard to get there.

[infinisil]

Opened a draft PR to track further development: #4154

[stale]

I marked this as stale due to inactivity. → More info

[lf-]

One thing I wanted this for is extracting data from overlays in order to implement some function on the overlay. Specifically, I wanted to gather all the names and versions of callHackage calls, in order to do an optimization. But this is not possible without loading the overlay twice, the first time with stubbed functions. So we just did that.

[anka-213]

Another use-case: Using .passthru attrs in IFD derivations without having to build the derivation first. For example, with callCabal2nix, you can get the inner derivation with (callCabal2nix "foo" ./. {}).passthru.cabal2nixDeriver, but without lazy attribute names nix will first build the cabal2nix derivation and only after that allow you to see what the derivation was.

Similarly for haskell.nix where the derivation for materializing a derivation is hidden inside a .passthru for a derivation that depends on that materialization, which leads to a catch-22, where the only way to access the generated code is to copy-paste from an error message (as is documented in the link above).

Now, both of these would be solvable by putting both derivations inside an outer record ({passthru = ...; main = ...;}), but that would be less convenient to use and would be a very breaking change.

---

I believe it could also make it easier to avoid infinite loops while bootstrapping, by allowing things like stdenv.cc.isGNU to not actually depend on cc.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/stability-of-function-return-values-in-nix/45124/1