Fine-grained impurity

[figsoda]

Is your feature request related to a problem? Please describe.

I found myself needing to use --impure with flakes just to give it access to NIXPKGS_ALLOW_UNFREE or the current directory. This is fine, but this is dangerous with untrusted nix code, and having to use --impure with flakes just doesn't feel good.

Describe the solution you'd like

Not everything has to be implemented, but this is what I imagine it would look like

nix build --expr "..." \
    --impure-for currentSystem # allow access to `builtins.currentSystem`
    --impure-for env:NIXPKGS_ALLOW_UNFREE=1 \ # allow evaluating unfree packages
    --impure-for env:NIXPKGS_ALLOW_UNFREE \ # pass through $NIXPKGS_ALLOW_UNFREE
    --impure-for ./. \ # allow access to the current directory
    --impure-for flake:nixpkgs # allow access to `builtins.getFlake "nixpkgs"`

Describe alternatives you've considered

Additional context

Not sure if this is a duplicate, because I don't know what to search in the issue tracker.

Priorities

Add 👍 to issues you find important.

[domenkozar]

Hell yeah 🙌

We need access to $PWD in devenv, as many things need to be placed relative to the project root.

I was thinking we could implement these rather declaratively, since the project most likely won't work without them.

Something like:

{
  inputs.pwd.url = "builtin:pwd";

  outputs = { pwd }: { 
  };
}

If someone would like implement this, please talk to me I'd sponsor the development.

[samueldr]

I think exposing those as impurities is a lost cause from the start.

Instead, they should be tracked as inputs. They are not impurities, they are CLI-declared inputs. Opting-in into an impurity makes them not an impurity, but a thing to track as an input.

Maybe --uses builtins:currentSystem and --uses env:PWD would help. And the same idea might even be usable within entirely declarative usage.

E.g. once --uses env:PWD is used, a new input for the current value of $PWD is intrinsically tracked. Similarly, --uses builtins:currentSystem means that there is an input that needs to be tracked, where it's the currentSystem. None of those are things that would change during a single evaluation, and really, most of these will not change across different evaluations within a given context.

---

I'm spitballing a bit here, but let's see if that helps thinking about the issue.

For declarative usage, a new builtins could be used, builtins.uses [...], and required to be at the root of an eval for declared uses. Any further uses found during eval, and listing new values not at the root would be an error.

That makes it so one of the goals (if I understand right) around pure evals, caching of evals, can still work since the given eval will already have had its inputs (the new uses) handled, and no new inputs than those at the root are allowed.

That's me thinking outside of the flakes declarative interface. For the flakes declarative interface, the same rules could apply to a uses root property, I guess.

[abathur]

Hopefully this isn't too abstract, but: when (i.e., what combination of eval-time, build-time, and run-time) are the ~impure parts of these various impurities needed? And can they all be ~frozen, as @samueldr imagines?

If, for example, we could put the impure parts in a box and create a pure reference to the box, when would people need to open the boxes?

In #8192 I ask about a flake-agnostic mechanism for working with/around impure system executables, and one way of describing the very vague sketch I made there is: a way to create pure references at eval time for impurities in cases where we don't need to open the box until runtime. I feel like these are at least adjacent if not related?

[samueldr]

For the discussion, I only thought about frozen and "trivial" values. Nothing like punching a "wormhole" across the store like your requirements are.

I would assume without thinking more than a few seconds that these would be distinct problems to solve likely in a different manner.

Your needs affect derivations in a more fundamental way, e.g. the derivation need to somehow carry that knowledge around.

The trivial eval inputs, in how I imagined them, are "just" values passed as fancy args, and evaluate as if your would have written them in the evaluated Nix code. Once evaluation is over, there is no moral difference between those inputs and just other Nix code.

BUT, other peeps, do not let my thoughts hold too much weight. Those are like, my opinions, peeps.

[abathur]

Even though I'm wondering about meeting some fraction of these impure needs without breaking eval/build-time purity, I want to clarify that I agree with a granular impurity mechanism along the lines of what figsoda suggests. I see the specificity this request would enable as deeply aligned with the Nix way.

I would assume without thinking more than a few seconds that these would be distinct problems to solve likely in a different manner.

I agree the solutions are at least partially distinct. Chiming in just in case it shakes out a more general theory of the kinds of impurity that are tractable, and because I suspect parts of the solutions could overlap and be generalized.

Your needs affect derivations in a more fundamental way, e.g. the derivation need to somehow carry that knowledge around.

The trivial eval inputs, in how I imagined them, are "just" values passed as fancy args, and evaluate as if your would have written them in the evaluated Nix code. Once evaluation is over, there is no moral difference between those inputs and just other Nix code.

Right. I guess this corresponds to opening the impurity box ~at (or immediately before) eval time and freezing that value as a flake input. It's a simpler approach, but I'm interpreting it as limited to flakes?

Maybe Domen's example clarifies why I suspect there's overlap.

Domen needs $PWD, but the environment variable is just a way of getting at what he really wants--a reference to the current directory. I'm not deeply familiar with devenv internals, but I skimmed/searched a bit and all of the cases I've seen so far look like they wouldn't actually need to open the "impurity box" until runtime.

I don't know if he'll agree, but it seems plausible that Domen's use-case could work with eval-time access to a deterministic absolute path which is a symlink to the current directory (usable at runtime but illegible within the build sandbox). This should be compatible with flakes as well?

Symlinks obviously won't cut it if someone needs runtime access to a non-path environment value. I guess envs could be handled with a realisation-time mechanism similar to the placeholder builtin and subsequent rewriting of the placeholders, but that couldn't work on the actual output. Maybe an uncached wrapper that just does placeholder rewriting after the input is realised? It's probably also fine if envs remain impure, especially with a granular mechanism for specifying them. (edit: Egh. I guess rewriting non-path values in binaries would also pose problems that we can't work around with padding.)

Anyways, I realize my sketch has a "draw the rest of the owl" problem when it comes to fixing these up at/around realisation. I'm not sure it can overcome the problems it would cause for people reusing outputs on systems that won't ever directly invoke Nix (though I guess the frozen input approach has a mirror-image of this problem where the literal values embedded could be inappropriate if used elsewhere).

[samueldr]

Right. I guess this corresponds to opening the impurity box ~at (or immediately before) eval time and freezing that value as a flake input. It's a simpler approach, but I'm interpreting it as limited to flakes?

Not Flakes, but pure evals. You can do pure evals without Flakes, too.

but it seems plausible that Domen's use-case could work with eval-time access to a deterministic absolute path which is a symlink to the current directory

Maybe.

I believe that for Domen's use-case, it's to build scripts with reference to the "pwd" of the evaluated project, so the environment built by devenv can refer to the project root. But don't assume I'm right. If my understanding is right, it's still a basic trivial value, not used to copy content to the store, nor refer to directory contents. If my understanding is right, the only thing that matters at eval time is the actual string of the path, and changing the directory wouldn't cause a different build.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/limited-interface-to-system-nix-external-dependencies/25825/10

[infinisil]

I've worked a bit on a draft RFC to allow pure evaluation and cached evaluation as builtins, which I think has really good potential. It's mostly a braindump for now, but the general idea should be glean-able, issues and PR's appreciated: https://github.com/tweag/epcb. I may be able to invest significantly more time into this

[roberth]

I agree with the general sentiment that impurities are inputs of some kind. However, I don't think turning them into flake inputs is the right solution. I'll explain with @domenkozar's example of passing a workdir to the devShell(s).

It's worth noting that this impurity only (or mostly) applies to nix run and nix develop.

These are some possible solutions I can think of:

Flake input, which is an error when unsupported

pwd is a flake input, and when it's used during the evaluation of nix build, it errors, while it does not for nix develop or nix run (perhaps. Maybe nix run should have a sister nix dev for development purposes)

Function

Make devShells a function, so we have e.g. devShells = { projectRoot, workingDirectory, ... }: { "x86_64-linux".default = mkShell ...; };.

This makes it abundantly clear that if you want to use such a dev shell in a context that isn't nix develop, the burden is on you to provide a pure projectRoot, or not call the function at all.

Similar solution can apply to nix run or nix dev (dev-specific run) or devApps (dev-specific run attributes).

Generalized flake configuration

Something along the lines of #6583 although I think making it per-derivation is the wrong angle. Also shells aren't even proper derivations, but that's another topic.

An alternative generalization, more global, per-flake configuration suffers from the same problem as the flake input idea.

---

In this case I think we should prefer the Function choice.

[DrRuhe]

I think something like this might be a great fit for enabling arguments that can be passed to flakes like discussed here.

[Alper-Celik]

adding some potential use cases:

• home manger needs to know your home dir and username to use it in standalone mode it would be usefull to have that as input of your flake too
• when using home manger and you want to create out of store symlink to file in current folder in flake based setup you need to hardcode path of file inside flake like

  {lib,...}:{
  xdg.configFile."<some file or folder path relative to $HOME>".source = lib.file.mkOutOfStoreSymlink "<config dir of your config>/<some file or folder>";
  }

  you need to at least hardcode your configs location to get files path it would be nice to get real directiory of configration file as an input.

[drupol]

This would be very useful for me. Basically I published a flake that install a developer's profile for the current user.

I added a note in the README explaining why the --impure flag is required: https://code.europa.eu/ecphp/devs-profile#note-about-impure-flag

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/devenv-1-0-rewrite-in-rust/41891/9

[SomeoneSerge]

--impure-for ./. \ # allow access to the current directory
...
Instead, they should be tracked as inputs.

I'll note that I had a loosely similar motivation in NixOS/nixpkgs#256230; the way this "declares" impure paths as "inputs" is using requiredSystemFeatures, which is admittedly limited and implicit