lib/types: Introduce lazyAttrsOf

[infinisil]

Motivation for this change

This introduces lazyAttrsOf, which is like attrsOf, but lazy in its values. So a single attribute can be accessed without forcing all others.

The tradeoff is that it has worse support for conditional definitions, so when defining foo.some-attr = mkIf false 10 for some lazyAttrsOf option foo, evaluating foo ? some-attr will be true even though it shouldn't be because of the mkIf false. So this option should ideally only be used where conditional definitions aren't needed.

I saw this originally in @oxij's #57123, where I thought the justification wasn't very convincing. But since then I found 2 other uses for such a type:

• Described in lib/types: Introduce lazyAttrsOf #70138 (comment). In short: By making _module.args be lazy in its values, we can evaluate parts of NixOS configurations without forcing all arguments. Conditional definitions are completely useless for module arguments, so this has no downside. This PR implements this.
• Described in [RFC 0042] NixOS settings options rfcs#42 (comment). In short: It allows lazyAttrsOf values to depend on other values in the same lazyAttrsOf without having infinite recursion. This is very useful in case absent values are encoded with nullOr, in which case conditional definitions can be replaced with assigment of null.

Best reviewed commit-by-commit.

TODO

• Write test
• Ensure backwards compatibility
• Write docs, mentioning what the drawback of this is
• Write comments and commit subject

cc @oxij @nbp @Profpatsch @rycee

[infinisil]

Wow! @arcnmx asked why setting _module.args.bar = throw "Bar evaluated!" would throw even when bar wasn't supposed to be evaluated. Here's a reproducible example:

with import <nixpkgs/lib>;
(evalModules {
  modules = [
    ({ foo, bar, ... }: {
      config._module.args = {
        foo = "Value of foo";
        bar = throw "Bar evaluated!";
      };
    })
  ];
}).config._module.args.foo

Well turns out args is defined as an attrsOf! With lazyAttrsOf this is no problem anymore, and I think this can be the first justified use of lazyAttrsOf in nixpkgs (especially since it doesn't make sense to have mkIf false on such a value).

[domenkozar]

Nice, that would replace our https://github.com/hercules-ci/nix-pre-commit-hooks/blob/master/nix/lazyAttrsOf.nix

[roberth]

mkIf false

I don't think this needs to be part of this PR, but I'd like to share my thoughts.

We could add an 'empty' or 'placeholder' value to the type in cases where mkIf false ... can be assigned a sane value. It is up to the consuming code to handle this case correctly though.
It requires some care from the module author though, because the 'empty' object must be treated the same as an attribute that doesn't exist.

In my implementation (thanks Domen for linking) I wrote a single generic function that can do both. In hindsight that approach complicated the code unnecessarily. If it's implemented it should probably be a separate type lazyAttrsWithDefault emptyValue t.

I wrote it that way to make the type to match the capability of attrsOf as closely as possible but I'm not certain that it's a useful niche to fill.

[roberth]

Suggested change

	joined attribute set. This is the lazy version of <varname>types.attrsOf
	merged attribute set where duplicate attributes are merged according to <replaceable>t</replaceable>. This is a lazy version of <varname>types.attrsOf

[roberth]

The documentation should point out that the type checking of all attributes still happens.
In my implementation for example it doesn't check the attributes, because my goal was to improve performance.

To replicate that behavior we could add type functions unchecked and lazyChecked which wrap a type to disable the check or run it before returning the value. That way you can have attrsOf (lazyChecked package) to prevent instantiating some packages if you don't need them, but still want to prevent non-package things from being used there.
These functions don't need to block this pr though.

[infinisil]

@roberth After thinking about it for a while, what you suggested with a default value sounds like a good idea. Because how this PR is now, there's no way to recover from a mkIf false, since the.option ? foo returns true but accessing it errors. I think we could do with a much simpler interface though, even going as far as using null as that special value without being able to customize it. Because I think all such attrsOf-like options should allow unsetting keys with null, so there one shouldn't need to distinguish between null and mkIf false. This however also implies that the given elemType needs to be nullOr, which leads to this nicely sounding type:

let {
  nullableAttrsOf = elemType:
    /* Implementation behaves like `lazyAttrsOf (nullOr elemType)`
       description could be `attribute set of (null for unset or elemType)`
    */;
}

This would work really well for NixOS/rfcs#42 too. What do you think of this?

[roberth]

I like the simplicity of "just null it", but it has the problem you mentioned. We can rely on the type parameter to provide the answer.
It removes the need for nullableAttrsOf, which I find to be non-obvious and potentially ambiguous as to which part can be null: the whole attrset, or the attribute value?

[roberth]

This could ask type for a placeholder value; something like

Suggested change

	else throw "The option `${showOption loc}' is used but not defined.";
	else t.placeholder or (throw "The option `${showOption loc}' is used but not defined.");

nullOr can then have { placeholder = null; }.

Not too complicated I think, while giving anyone the opportunity to add placeholders as appropriate for their use cases.

[jtojnar]

Do not forget to add the synchronization comment from valueDefined.

[infinisil]

Done that

[infinisil]

@roberth I updated the code with your suggestion. I've decided to use emptyValue to represent such an empty value, since placeholder is already a builtin nix function name, and it really represents an empty value. See commit df5478f. I updated the tests and the documentation on this too. Let me know if this looks good to you

[infinisil]

Rebased to fix conflicts

[infinisil]

@roberth Applied your first 2 suggestions

[infinisil]

I noticed that I didn't implement type merging, so I did that now. E.g. this now works (foo.one.bar can be evaluated without an exception):

{ config, lib, pkgs, ... }: {

  imports = [
    {
      options.foo = lib.mkOption {
        type = lib.types.lazyAttrsOf (lib.types.submodule {
          options.bar = lib.mkOption {
            default = false;
          };
        });
      };
    }
    {
      options.foo = lib.mkOption {
        type = lib.types.lazyAttrsOf (lib.types.submodule {
          options.baz = lib.mkOption {
            default = false;
          };
        });
      };
    }
  ];

  config = {
    boot.loader.grub.device = "nodev";
    fileSystems."/".device = "x";

    foo = {
      one.bar = true;
      two = throw "nothing";
    };
  };
}

Using attrsOf elemType // wasn't worth it anymore, and it also didn't seem to harmonize with type merging well at all, so I'm just declaring all attributes separately in lazyAttrsOf.

[infinisil]

Rebased to fix conflicts, will merge after ofborg is happy

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/january-2020-in-nixos/5771/1