Environment Setup Fails Due to Missing .devenv.flake.nix after devenv update

[gonzaloetjo]

Describe the bug
I had a working devenv with an overlay for foundry, but it started failing once I did devenv update (from 0.6.3 to 1.0.3).

The process fails due to an error related to missing files and possible path issues. The specific error message indicates that the .devenv.flake.nix file cannot be found (despite it being there), leading to a failure in loading the environment. Here's the specific part of the error log that highlights the issue:

direnv: loading ~/devenv/foundry-devenv/.envrc
direnv: loading https://raw.githubusercontent.com/cachix/devenv/d1f7b48e35e6dee421cfd0f51481d17f77586997/direnvrc
direnv: using devenv
error: opening file '/nix/store/0n86420hx4bydkpx8q7a4vqw46f5ihhp-source/.devenv.flake.nix': No such file or directory
...
...
direnv: loading the environment failed
direnv: error exit status 1

.rw-r--r--  5.1k getse 17 Apr 11:13  ├── .devenv.flake.nix
drwxr-xr-x     - getse 21 Aug  2023  ├── .direnv
.rw-r--r--   185 getse 17 Apr 11:09  ├── .envrc
drwxr-xr-x     - getse 17 Apr 11:12  ├── .git
drwxr-xr-x     - getse 21 Aug  2023  │  ├── branches
.rw-r--r--    14 getse 15 Apr 16:48  │  ├── COMMIT_EDITMSG
.rw-r--r--   266 getse 21 Aug  2023  │  ├── config
.rw-r--r--    73 getse 21 Aug  2023  │  ├── description
.rw-r--r--    97 getse 21 Aug  2023  │  ├── FETCH_HEAD
.rw-r--r--    21 getse 21 Aug  2023  │  ├── HEAD
drwxr-xr-x     - getse 21 Aug  2023  │  ├── hooks
.rw-r--r--  2.7k getse 17 Apr 11:12  │  ├── index
drwxr-xr-x     - getse 21 Aug  2023  │  ├── info
drwxr-xr-x     - getse 21 Aug  2023  │  ├── logs
drwxr-xr-x     - getse 15 Apr 16:48  │  ├── objects
.rw-r--r--    41 getse 21 Aug  2023  │  ├── ORIG_HEAD
drwxr-xr-x     - getse 21 Aug  2023  │  └── refs
.rw-r--r--    53 getse 21 Aug  2023  ├── .gitignore
.rw-r--r--  8.9k getse 17 Apr 11:12  ├── devenv.lock
.rw-r--r--  2.0k getse 21 Aug  2023  ├── devenv.nix
.rw-r--r--   281 getse 21 Aug  2023  ├── devenv.yaml
drwxr-xr-x     - getse 21 Aug  2023  ├── foundry-overlay
.rw-r--r--  2.8k getse 21 Aug  2023  │  ├── flake.lock
.rw-r--r--   225 getse 21 Aug  2023  │  └── flake.nix
.rw-r--r--   11k getse 21 Aug  2023  ├── LICENSE
.rw-r--r--   781 getse 21 Aug  2023  ├── README.md

To reproduce

Please find the necessary files (devenv.nix, devenv.yaml, and devenv.lock) and the complete error logs in this https://gist.github.com/gonzaloetjo/41b62b2a97387b8b8029ba3c2fdd5d8c.

Expected behavior
The environment should load without errors, sourcing all necessary Nix packages and configurations. Previously, this setup loaded correctly without any issues.

Version

$ devenv version
1.0.3

Previously, this configuration was working with version 0.6.3.

The repository for the current environment setup is found here.

[mcdonc]

Here is an easier repro:

[chrism@optinix:~/projects]$ git clone git@github.com:gonzaloetjo/foundry-devenv.git
Cloning into 'foundry-devenv'...
remote: Enumerating objects: 34, done.
remote: Counting objects: 100% (34/34), done.
remote: Compressing objects: 100% (19/19), done.
remote: Total 34 (delta 15), reused 29 (delta 13), pack-reused 0
Receiving objects: 100% (34/34), 13.43 KiB | 764.00 KiB/s, done.
Resolving deltas: 100% (15/15), done.


[chrism@optinix:~/projects]$ nix profile install --accept-flake-config github:cachix/devenv/v1.0.3

[chrism@optinix:~/projects]$ cd foundry-devenv/

[chrism@optinix:~/projects/foundry-devenv]$ git log --oneline|head -1
352f845 feat: update


[chrism@optinix:~/projects/foundry-devenv]$ devenv shell
• Building shell ...
• Using Cachix: devenv
error:
       … while calling the 'derivationStrict' builtin
         at <nix/derivation-internal.nix>:9:12:
            8|
            9|   strict = derivationStrict drvAttrs;
             |            ^
           10|

       … while evaluating derivation 'devenv-shell'
         whose name attribute is located at «github:NixOS/nixpkgs/2748d22b45a99fb2deafa5f11c7531c212b2cefa»/pkgs/stdenv/generic/make-derivation.nix:331:7

       … while evaluating attribute 'DEVENV_PROFILE' of derivation 'devenv-shell'

       … while evaluating derivation 'devenv-profile'
         whose name attribute is located at «github:NixOS/nixpkgs/2748d22b45a99fb2deafa5f11c7531c212b2cefa»/pkgs/stdenv/generic/make-derivation.nix:331:7

       … while evaluating attribute 'passAsFile' of derivation 'devenv-profile'
         at «github:NixOS/nixpkgs/2748d22b45a99fb2deafa5f11c7531c212b2cefa»/pkgs/build-support/trivial-builders/default.nix:69:9:
           68|         inherit buildCommand name;
           69|         passAsFile = [ "buildCommand" ]
             |         ^
           70|           ++ (derivationArgs.passAsFile or [ ]);

       … from call site
         at «github:cachix/devenv/a7939d53e20cd2f44b209fac0aaa5389a9078127»/src/modules/top-level.nix:15:13:
           14|     name = "devenv-profile";
           15|     paths = lib.flatten (builtins.map drvOrPackageToPaths config.packages);
             |             ^
           16|     ignoreCollisions = true;

       … while calling 'flatten'
         at «github:NixOS/nixpkgs/2748d22b45a99fb2deafa5f11c7531c212b2cefa»/lib/lists.nix:383:13:
          382|   */
          383|   flatten = x:
             |             ^
          384|     if isList x

       … from call site
         at «github:cachix/devenv/a7939d53e20cd2f44b209fac0aaa5389a9078127»/src/modules/top-level.nix:15:59:
           14|     name = "devenv-profile";
           15|     paths = lib.flatten (builtins.map drvOrPackageToPaths config.packages);
             |                                                           ^
           16|     ignoreCollisions = true;

       … while calling anonymous lambda
         at «github:NixOS/nixpkgs/2748d22b45a99fb2deafa5f11c7531c212b2cefa»/lib/attrsets.nix:1171:18:
         1170|         mapAttrs
         1171|           (name: value:
             |                  ^
         1172|             if isAttrs value && cond value

       … from call site
         at «github:NixOS/nixpkgs/2748d22b45a99fb2deafa5f11c7531c212b2cefa»/lib/attrsets.nix:1174:18:
         1173|             then recurse (path ++ [ name ]) value
         1174|             else f (path ++ [ name ]) value);
             |                  ^
         1175|     in

       … while calling anonymous lambda
         at «github:NixOS/nixpkgs/2748d22b45a99fb2deafa5f11c7531c212b2cefa»/lib/modules.nix:242:72:
          241|           # For definitions that have an associated option
          242|           declaredConfig = mapAttrsRecursiveCond (v: ! isOption v) (_: v: v.value) options;
             |                                                                        ^
          243|

       … while evaluating the option `packages':

       … from call site
         at «github:NixOS/nixpkgs/2748d22b45a99fb2deafa5f11c7531c212b2cefa»/lib/modules.nix:846:59:
          845|       if isDefined then
          846|         if all (def: type.check def.value) defsFinal then type.merge loc defsFinal
             |                                                           ^
          847|         else let allInvalid = filter (def: ! type.check def.value) defsFinal;

       … while calling 'merge'
         at «github:NixOS/nixpkgs/2748d22b45a99fb2deafa5f11c7531c212b2cefa»/lib/types.nix:552:20:
          551|       check = isList;
          552|       merge = loc: defs:
             |                    ^
          553|         map (x: x.value) (filter (x: x ? value) (concatLists (imap1 (n: def:

       … while calling anonymous lambda
         at «github:NixOS/nixpkgs/2748d22b45a99fb2deafa5f11c7531c212b2cefa»/lib/types.nix:553:35:
          552|       merge = loc: defs:
          553|         map (x: x.value) (filter (x: x ? value) (concatLists (imap1 (n: def:
             |                                   ^
          554|           imap1 (m: def':

       … from call site
         at «github:NixOS/nixpkgs/2748d22b45a99fb2deafa5f11c7531c212b2cefa»/lib/types.nix:553:38:
          552|       merge = loc: defs:
          553|         map (x: x.value) (filter (x: x ? value) (concatLists (imap1 (n: def:
             |                                      ^
          554|           imap1 (m: def':

       … while calling anonymous lambda
         at «github:NixOS/nixpkgs/2748d22b45a99fb2deafa5f11c7531c212b2cefa»/lib/lists.nix:334:29:
          333|   */
          334|   imap1 = f: list: genList (n: f (n + 1) (elemAt list n)) (length list);
             |                             ^
          335|

       … from call site
         at «github:NixOS/nixpkgs/2748d22b45a99fb2deafa5f11c7531c212b2cefa»/lib/lists.nix:334:32:
          333|   */
          334|   imap1 = f: list: genList (n: f (n + 1) (elemAt list n)) (length list);
             |                                ^
          335|

       … while calling anonymous lambda
         at «github:NixOS/nixpkgs/2748d22b45a99fb2deafa5f11c7531c212b2cefa»/lib/types.nix:554:21:
          553|         map (x: x.value) (filter (x: x ? value) (concatLists (imap1 (n: def:
          554|           imap1 (m: def':
             |                     ^
          555|             (mergeDefinitions

       … while calling anonymous lambda
         at «github:NixOS/nixpkgs/2748d22b45a99fb2deafa5f11c7531c212b2cefa»/lib/modules.nix:824:28:
          823|         # Process mkMerge and mkIf properties.
          824|         defs' = concatMap (m:
             |                            ^
          825|           map (value: { inherit (m) file; inherit value; }) (builtins.addErrorContext "while evaluating definitions from `${m.file}':" (dischargeProperties m.value))

       … while evaluating definitions from `/nix/store/virtual0000000000000000000000006-source/devenv.nix':

       … from call site
         at «github:NixOS/nixpkgs/2748d22b45a99fb2deafa5f11c7531c212b2cefa»/lib/modules.nix:825:137:
          824|         defs' = concatMap (m:
          825|           map (value: { inherit (m) file; inherit value; }) (builtins.addErrorContext "while evaluating definitions from `${m.file}':" (dischargeProperties m.value))
             |                                                                                                                                         ^
          826|         ) defs;

       … while calling 'dischargeProperties'
         at «github:NixOS/nixpkgs/2748d22b45a99fb2deafa5f11c7531c212b2cefa»/lib/modules.nix:896:25:
          895|   */
          896|   dischargeProperties = def:
             |                         ^
          897|     if def._type or "" == "merge" then

       … from call site
         at «github:shazow/foundry.nix/ece7c960a440c6725a7a5576d1f49a5fabde3747»/flake.nix:18:23:
           17|         pkgs = import nixpkgs { inherit system; };
           18|         foundry-bin = import ./foundry-bin { inherit pkgs; };
             |                       ^
           19|         # TODO: Add a source-based derivation someday

       … while calling anonymous lambda
         at «github:shazow/foundry.nix/ece7c960a440c6725a7a5576d1f49a5fabde3747»/foundry-bin/default.nix:1:1:
            1| {pkgs ? import <nixpkgs> {}}: let
             | ^
            2|   inherit (pkgs) stdenv lib;

       … while calling anonymous lambda
         at «nix-internal»/call-flake.nix:81:25:
           80|           inputs = builtins.mapAttrs
           81|             (inputName: inputSpec: allNodes.${resolveInput inputSpec})
             |                         ^
           82|             (node.inputs or {});

       … from call site
         at «nix-internal»/call-flake.nix:81:36:
           80|           inputs = builtins.mapAttrs
           81|             (inputName: inputSpec: allNodes.${resolveInput inputSpec})
             |                                    ^
           82|             (node.inputs or {});

       … while calling anonymous lambda
         at «nix-internal»/call-flake.nix:39:13:
           38|     builtins.mapAttrs
           39|       (key: node:
             |             ^
           40|         let

       error:
       error: opening file '/nix/store/0n86420hx4bydkpx8q7a4vqw46f5ihhp-source/.devenv.flake.nix': No such file or directory

✖ Command produced the following output:


✔ Building shell in 1.1s.
Error:   × Command `/nix/store/16z8qb8j04nzahvxgms9wn9qba1fjh4z-nix-devenv-
  │ 2.21.0pre20240315_c5bbf14/bin/nix --show-trace --extra-experimental-
  │ features nix-command --extra-experimental-features flakes --option
  │ warn-dirty false --option eval-cache false --keep-going --max-
  │ jobs 3 print-dev-env --profile /home/chrism/projects/foundry-
  │ devenv/.devenv/gc/shell --option extra-substituters https://
  │ devenv.cachix.org --option extra-trusted-public-keys devenv.cachix.org-
  │ 1:w1cLUi8dv3hnoSPGAuibQv+f9TZLr6cv/Hm9XgU50cw= nixpkgs-python.cachix.org-
  │ 1:hxjI7pFxTyuTHn2NkvWCrAUcNZLNS3ZAvfYNuYifcEU=` failed with with exit
  │ code 1

[mcdonc]

The nix path its looking in is some version of nixpkgs.

[chrism@optinix:~/projects/foundry-devenv]$ ls /nix/store/0n86420hx4bydkpx8q7a4vqw46f5ihhp-source
CONTRIBUTING.md  default.nix  flake.nix  maintainers  pkgs
COPYING          doc          lib        nixos        README.md

[mcdonc]

Removing the dependency on foundry-bin allows for a shell:

[chrism@optinix:~/projects/foundry-devenv]$ git diff
diff --git a/devenv.nix b/devenv.nix
index ccdddcb..69c034f 100644
--- a/devenv.nix
+++ b/devenv.nix
@@ -15,7 +15,7 @@ in {
 
     # https://devenv.sh/packages/
     packages = [
-      pkgs.foundry-bin
+#      pkgs.foundry-bin
       pkgs.git
       pkgs.jq
       pkgs.zip

[chrism@optinix:~/projects/foundry-devenv]$ devenv shell
• Building shell ...
• Using Cachix: devenv
✔ Building shell in 10.8s.
• Entering shell
✨ devenv 1.0.3 is out of date. Please update to 1.0.4: https://devenv.sh/getting-started/#installation
*********************************************************
*                                                       *
*    Welcome to the Nix dev environment for foundry!    *
*                                                       *
*    For more information, please visit:                *
*    https://github.com/shazow/foundry.nix              *
*    https://github.com/cachix/devenv                   *
*                                                       *
*********************************************************
(devenv) 

foundry-bin is available only as a result of the following overlay:

inputs:
  ...
  foundry-overlay:
    url: path:./foundry-overlay
    overlays:
      - default

{
  inputs = {
    foundry.url = "github:shazow/foundry.nix/monthly";
  };

  outputs = { self, foundry, ... }: {
    overlays.default = self: super: {
      foundry-bin = foundry.defaultPackage.${self.system};
    };
  };
}

Does not appear to be related to being an overlay, because if I change things around such that we just use it directly:

inputs:
  ...
  foundry:
    url: github:shazow/foundry.nix/monthly

and

{ inputs, pkgs, ... }:
{
   ....
   packages = [
      inputs.foundry.defaultPackage.${pkgs.system}
     ...
    ]
}

We wind up with more or less the same traceback.

[mcdonc]

The simplest possible repro of the issue is this (my foundry-devenv repo fork "simplify-bug-repro" branch has a devenv.nix that incudes only the remote flake as a package):

[chrism@optinix:~/projects]$ git clone git@github.com:mcdonc/foundry-devenv.git
Cloning into 'foundry-devenv'...
remote: Enumerating objects: 39, done.
remote: Counting objects: 100% (39/39), done.
remote: Compressing objects: 100% (23/23), done.
remote: Total 39 (delta 16), reused 34 (delta 14), pack-reused 0
Receiving objects: 100% (39/39), 14.05 KiB | 1.28 MiB/s, done.
Resolving deltas: 100% (16/16), done.

[chrism@optinix:~/projects]$ cd foundry-devenv/

[chrism@optinix:~/projects/foundry-devenv]$ git checkout simplify-bug-repro 
branch 'simplify-bug-repro' set up to track 'origin/simplify-bug-repro'.
Switched to a new branch 'simplify-bug-repro'

[chrism@optinix:~/projects/foundry-devenv]$ devenv shell
• Building shell ...
• Using Cachix: devenv
error:
       … while calling the 'derivationStrict' builtin
         at <nix/derivation-internal.nix>:9:12:
...

           38|     builtins.mapAttrs
           39|       (key: node:
             |             ^
           40|         let

       error:
       error: opening file '/nix/store/0n86420hx4bydkpx8q7a4vqw46f5ihhp-source/.devenv.flake.nix': No such file or directory

✖ Command produced the following output:


✔ Building shell in 1.1s.
Error:   × Command `/nix/store/16z8qb8j04nzahvxgms9wn9qba1fjh4z-nix-devenv-
  │ 2.21.0pre20240315_c5bbf14/bin/nix --show-trace --extra-experimental-
  │ features nix-command --extra-experimental-features flakes --option
  │ warn-dirty false --option eval-cache false --keep-going --max-jobs 3
  │ print-dev-env --profile /home/chrism/projects/foundry-devenv/.devenv/
  │ gc/shell --option extra-substituters https://devenv.cachix.org
  │ --option extra-trusted-public-keys nixpkgs-python.cachix.org-
  │ 1:hxjI7pFxTyuTHn2NkvWCrAUcNZLNS3ZAvfYNuYifcEU= devenv.cachix.org-
  │ 1:w1cLUi8dv3hnoSPGAuibQv+f9TZLr6cv/Hm9XgU50cw=` failed with with exit
  │ code 1

When I attempt to put the exact same files that are currently in the simplify-bug-repro branch into a subdir of examples in devenv, devenv shell runs fine against them and produces a shell without the exception. So that's a clue. :)

[therealpxc]

This is the same issue that I ran into and mentioned to Domen on Matrix a few days ago. I've created a barebones repo to demonstrate/reproduce the issue here, which (optionally) depends on a very simple flake which can be found here.

This issue came up for me when I started work on a pre-commit hook that scans repos for secrets using TruffleHog, which at the time I started was broken on Darwin in Nixpkgs. I noticed issues with pulling in my fixed version of the TruffleHog package:

• in my own fork of Nixpkgs on GitHub
• in my own fork of Nixpkgs in a local checkout
• in an out-of-tree package from a private flake with various Nix packages on our GitLab

I've noticed a few workarounds in this:

1. Using my own fork of Nixpkgs to replace the default nixpkgs input works fine.
2. Using the sort of pre-flake method of importing Nixpkgs from my input, then grabbing its package works fine as long as flake: false is set in devenv.yaml for that input.

I've also noticed that the pre-commit hook I've defined actually works, even when devShell evaluation fails. That part evaluates fine.

I don't know how indicative it is, but the failure to find a .devenv.flake.nix file in my case seems to happen in one of devenv's own dependencies, namely flake-parts:

       … while calling anonymous lambda
         at «nix-internal»/call-flake.nix:39:13:
           38|     builtins.mapAttrs
           39|       (key: node:
             |             ^
           40|         let

       error:
       error: opening file '/nix/store/2hc9lg18zd6yabw9jqj0wy3s9kyvkzp0-source/.devenv.flake.nix': No such file or directory

✖ Command produced the following output:


Error:   × Command `/nix/store/as7cgg7qvnkamrcycbgcdvhs0kv7wphx-nix-2.21-devenv/bin/nix --show-trace --extra-experimental-features nix-command --extra-experimental-features flakes
  │ --option warn-dirty false --option eval-cache false --keep-going --max-jobs 5 print-dev-env --profile /Users/patcal04/Sandbox/empty-project/.devenv/gc/shell --option extra-
  │ substituters https://devenv.cachix.org --option extra-trusted-public-keys devenv.cachix.org-1:w1cLUi8dv3hnoSPGAuibQv+f9TZLr6cv/Hm9XgU50cw=` failed with with exit code 1

direnv: failed to build the devenv environment. devenv.nix may contain errors. see above.

And when I inspect that Nix store path, I can see it's flake-parts by the README:

# Flake Parts

_Core of a distributed framework for writing Nix Flakes._

`flake-parts` provides the options that represent standard flake attributes
and establishes a way of working with `system`.
Opinionated features are provided by an ecosystem of modules that you can import.

`flake-parts` _itself_ has the goal to be a minimal mirror of the Nix flake schema.```

[therealpxc]

The nix path its looking in is some version of nixpkgs.

[chrism@optinix:~/projects/foundry-devenv]$ ls /nix/store/0n86420hx4bydkpx8q7a4vqw46f5ihhp-source
CONTRIBUTING.md  default.nix  flake.nix  maintainers  pkgs
COPYING          doc          lib        nixos        README.md

It seems this varies. I've seen devenv mistakenly look for a .devenv.flake.nix in not just nixpkgs, but also flake-parts. I think maybe we're looking for a .devenv.flake.nix when we should still be looking for a flake.nix.

[therealpxc]

I suspect the most fruitful place to look next will be comparing the .devenv.flake.nix files generated by 0.x versions of devenv to those generated by 1.x versions, since that's where the shell is defined. Incidentally, the one I get with my example repo is this:

{
  inputs =
    let
      version = "1.0.4";
system = "aarch64-darwin";
devenv_root = "/Users/patcal04/Sandbox/empty-project";
devenv_dotfile = ./.devenv;
devenv_dotfile_string = ".devenv";
container_name = null;
devenv_tmpdir = "/var/folders/fw/73494rb94f7b5vppp_c7_c5wl2kfgf/T/";
devenv_runtime = "/var/folders/fw/73494rb94f7b5vppp_c7_c5wl2kfgf/T/devenv-50fa3f4";

        in {
        pre-commit-hooks.url = "github:cachix/pre-commit-hooks.nix";
      pre-commit-hooks.inputs.nixpkgs.follows = "nixpkgs";
      nixpkgs.url = "github:cachix/devenv-nixpkgs/rolling";
      devenv.url = "github:cachix/devenv?dir=src/modules";
      } // (if builtins.pathExists (devenv_dotfile + "/flake.json")
      then builtins.fromJSON (builtins.readFile (devenv_dotfile +  "/flake.json"))
      else { });

      outputs = { nixpkgs, ... }@inputs:
        let
          version = "1.0.4";
system = "aarch64-darwin";
devenv_root = "/Users/patcal04/Sandbox/empty-project";
devenv_dotfile = ./.devenv;
devenv_dotfile_string = ".devenv";
container_name = null;
devenv_tmpdir = "/var/folders/fw/73494rb94f7b5vppp_c7_c5wl2kfgf/T/";
devenv_runtime = "/var/folders/fw/73494rb94f7b5vppp_c7_c5wl2kfgf/T/devenv-50fa3f4";

            devenv =
            if builtins.pathExists (devenv_dotfile + "/devenv.json")
            then builtins.fromJSON (builtins.readFile (devenv_dotfile + "/devenv.json"))
            else { };
          getOverlays = inputName: inputAttrs:
            map
              (overlay:
                let
                  input = inputs.${inputName} or (throw "No such input `${inputName}` while trying to configure overlays.");
                in
                  input.overlays.${overlay} or (throw "Input `${inputName}` has no overlay called `${overlay}`. Supported overlays: ${nixpkgs.lib.concatStringsSep ", " (builtins.attrNames input.overlays)}"))
              inputAttrs.overlays or [ ];
          overlays = nixpkgs.lib.flatten (nixpkgs.lib.mapAttrsToList getOverlays (devenv.inputs or { }));
          pkgs = import nixpkgs {
            inherit system;
            config = {
              allowUnfree = devenv.allowUnfree or false;
              allowBroken = devenv.allowBroken or false;
              permittedInsecurePackages = devenv.permittedInsecurePackages or [ ];
            };
            inherit overlays;
          };
          lib = pkgs.lib;
          importModule = path:
            if lib.hasPrefix "./" path
            then if lib.hasSuffix ".nix" path
            then ./. + (builtins.substring 1 255 path)
            else ./. + (builtins.substring 1 255 path) + "/devenv.nix"
            else if lib.hasPrefix "../" path
            then throw "devenv: ../ is not supported for imports"
            else
              let
                paths = lib.splitString "/" path;
                name = builtins.head paths;
                input = inputs.${name} or (throw "Unknown input ${name}");
                subpath = "/${lib.concatStringsSep "/" (builtins.tail paths)}";
                devenvpath = "${input}" + subpath;
                devenvdefaultpath = devenvpath + "/devenv.nix";
              in
              if lib.hasSuffix ".nix" devenvpath
              then devenvpath
              else if builtins.pathExists devenvdefaultpath
              then devenvdefaultpath
              else throw (devenvdefaultpath + " file does not exist for input ${name}.");
          project = pkgs.lib.evalModules {
            specialArgs = inputs // { inherit inputs pkgs; };
            modules = [
              (inputs.devenv.modules + /top-level.nix)
              {
                devenv.cliVersion = version;
                devenv.root = devenv_root;
                devenv.dotfile = devenv_root + "/" + devenv_dotfile_string;
              }
              (pkgs.lib.optionalAttrs (inputs.devenv.isTmpDir or false) {
                devenv.tmpdir = devenv_tmpdir;
                devenv.runtime = devenv_runtime;
              })
              (pkgs.lib.optionalAttrs (container_name != null) {
                container.isBuilding = pkgs.lib.mkForce true;
                containers.${container_name}.isBuilding = true;
              })
            ] ++ (map importModule (devenv.imports or [ ])) ++ [
              ./devenv.nix
              (devenv.devenv or { })
              (if builtins.pathExists ./devenv.local.nix then ./devenv.local.nix else { })
            ];
          };
          config = project.config;

          options = pkgs.nixosOptionsDoc {
            options = builtins.removeAttrs project.options [ "_module" ];
            # Unpack Nix types, e.g. literalExpression, mDoc.
            transformOptions =
              let isDocType = v: builtins.elem v [ "literalDocBook" "literalExpression" "literalMD" "mdDoc" ];
              in lib.attrsets.mapAttrs (_: v:
                if v ? _type && isDocType v._type then
                  v.text
                else if v ? _type && v._type == "derivation" then
                  v.name
                else
                  v
              );
          };
        in
        {
          packages."${system}" = {
            optionsJSON = options.optionsJSON;
            inherit (config) info procfileScript procfileEnv procfile;
            ci = config.ciDerivation;
          };
          devenv = config;
          devShell."${system}" = config.shell;
        };
      }

[mcdonc]

It seems this varies. I've seen devenv mistakenly look for a .devenv.flake.nix in not just nixpkgs, but also flake-parts. I think maybe we're looking for a .devenv.flake.nix when we should still be looking for a flake.nix.

That sounds likely, thanks for relating it!

[mcdonc]

There must be some implicit thing that is consuming .devenv.flake.nix because it's unreferenced in devenv's source other than creating it. Apologies if this is a FAQ but is there some feature of Nix itself that defers to things with some naming convention like this?

[therealpxc]

There must be some implicit thing that is consuming .devenv.flake.nix because it's unreferenced in devenv's source other than creating it. Apologies if this is a FAQ but is there some feature of Nix itself that defers to things with some naming convention like this?

Yes. I believe that's in Nix itself. Notice that the Nix path in our stack traces seems to be a devenv-specific fork. On my system, when I try to use nix repl in a devenv project, I get different results with the system Nix than with the one that appears in my stack traces. On my work machine (which uses Nix 2.21.2):

empty-project on  main [!⇡] on ☁️  (us-west-2) 
❯ nix repl .
Nix 2.21.2
Type :? for help.
warning: Git tree '/Users/patcal04/Sandbox/empty-project' is dirty
error: path '/nix/store/0ibm4njlhjdxmikwjpy5zzrh99wxnnlv-source/.devenv.flake.nix' does not exist

empty-project on  main [!⇡] on ☁️  (us-west-2) 
❯ /nix/store/as7cgg7qvnkamrcycbgcdvhs0kv7wphx-nix-2.21-devenv/bin/nix repl .
Nix 2.21.0
Type :? for help.
warning: Git tree '/Users/patcal04/Sandbox/empty-project' is dirty
warning: will not write lock file of flake 'git+file:///Users/patcal04/Sandbox/empty-project' because it has an unlocked input ('path:/Users/patcal04/Sandbox/nixpkgs')
Loading installable 'git+file:///Users/patcal04/Sandbox/empty-project#'...
Added 3 variables.
nix-repl> :help
The following commands are available:

  <expr>                       Evaluate and print expression
  <x> = <expr>                 Bind expression to variable
  :a, :add <expr>              Add attributes from resulting set to scope
  :b <expr>                    Build a derivation
  :bl <expr>                   Build a derivation, creating GC roots in the
                               working directory
  :e, :edit <expr>             Open package or function in $EDITOR
  :i <expr>                    Build derivation, then install result into
                               current profile
  :l, :load <path>             Load Nix expression and add it to scope
  :lf, :load-flake <ref>       Load Nix flake and add it to scope
  :p, :print <expr>            Evaluate and print expression recursively
  :q, :quit                    Exit nix-repl
  :r, :reload                  Reload all files
  :sh <expr>                   Build dependencies of derivation, then start
                               nix-shell
  :t <expr>                    Describe result of evaluation
  :u <expr>                    Build derivation, then start nix-shell
  :doc <expr>                  Show documentation of a builtin function
  :log <expr>                  Show logs for a derivation
  :te, :trace-enable [bool]    Enable, disable or toggle showing traces for
                               errors
  :?, :help                    Brings up this help menu

nix-repl> 

On my other machine, which I just did some testing on with my repos for reproduction this morning, I found a few other surprising things. (Post to follow, sorry in advance for double-posting.)

[therealpxc]

Check this out: 'stable' Nix (2.18.2) doesn't know how to open a flake repl in a devenv repo, but the Nix whose store path ends in -devenv that shows up in my stack traces does:

devenv-problem-example on  main [!⇡] 
❯ nix repl .
Welcome to Nix 2.18.2. Type :? for help.

path '/Users/pxc/Documents/Code/devenv-problem-example' does not contain a 'flake.nix', searching up
error: path '/Users/pxc/Documents/Code/devenv-problem-example' is not part of a flake (neither it nor its parent directories contain a 'flake.nix' file)

devenv-problem-example on  main [!⇡] 
❯ /nix/store/as7cgg7qvnkamrcycbgcdvhs0kv7wphx-nix-2.21-devenv/bin/nix repl .
Nix 2.21.0
Type :? for help.
warning: Git tree '/Users/pxc/Documents/Code/devenv-problem-example' is dirty
warning: will not write lock file of flake 'git+file:///Users/pxc/Documents/Code/devenv-problem-example' because it has an unlocked input ('path:/Users/pxc/Documents/Code/nixpkgs')
Loading installable 'git+file:///Users/pxc/Documents/Code/devenv-problem-example#'...
Added 3 variables.
nix-repl> :q 

As for where in Nix to look, our clue is of course right there in the stack traces:

       … while calling anonymous lambda
         at «nix-internal»/call-flake.nix:81:25:
           80|           inputs = builtins.mapAttrs
           81|             (inputName: inputSpec: allNodes.${resolveInput inputSpec})
             |                         ^
           82|             (node.inputs or {});

       … from call site
         at «nix-internal»/call-flake.nix:81:36:
           80|           inputs = builtins.mapAttrs
           81|             (inputName: inputSpec: allNodes.${resolveInput inputSpec})
             |                                    ^
           82|             (node.inputs or {});

       … while calling anonymous lambda
         at «nix-internal»/call-flake.nix:39:13:
           38|     builtins.mapAttrs
           39|       (key: node:
             |             ^

That's this file (permalink). I assume devenv patches this file somewhere to have it look for .devenv.flake.nix rather than just flake.nix

I assume there's also some other trickery, that devenv perhaps invokes call-flake.nix manually somehow, because this (from the nix repl in my immediately previous post):

error: path '/nix/store/0ibm4njlhjdxmikwjpy5zzrh99wxnnlv-source/.devenv.flake.nix' does not exist

is complaining about a missing .devenv.flake.nix in my example project, where of course it's present. It's missing in the Nix store because (per the devenv .gitignore defaults), it's excluded from version control and the normal mechanisms won't copy it in there.

At any rate, our issues are likely in the patched-up Nix that it seems devenv uses for evaluation, perhaps in the embedded Nix code for libexpr.

[mcdonc]

At any rate, our issues are likely in the patched-up Nix that it seems devenv uses for evaluation, perhaps in the embedded Nix code for libexpr.

Oh geez, well-spotted.

[therealpxc]

I assume there' also some other trickery, that devenv perhaps invokes call-flake.nix manually somehow, because this (from the nix repl in my immediately previous post):

error: path '/nix/store/0ibm4njlhjdxmikwjpy5zzrh99wxnnlv-source/.devenv.flake.nix' does not exist

is complaining about a missing .devenv.flake.nix in my example project, where of course it's present. It's missing in the Nix store because (per the devenv .gitignore defaults), it's excluded from version control and the normal mechanisms won't copy it in there.

Yep, check it out: one of the changes in Domen's fork of Nix is to copy untracked files into the store for flake repos: domenkozar/nix@138a986

Incidentally I think that tells us we might (also?) want to look at libfetchers. Here's a starting point for files of interest:

nix-devenv on  HEAD (b24a931) via C v15.0.0-clang 
❯ rg -F '.devenv.flake.nix'
src/libfetchers/git-utils.cc
311:        if (pathExists(cwd / ".devenv.flake.nix")) {
312:            info.files.insert(CanonPath((cwd / ".devenv.flake.nix").string()).removePrefix(CanonPath(path.string())).rel());

src/libexpr/flake/flakeref.cc
120:            if (!allowMissing && !pathExists(path + "/.devenv.flake.nix")){
121:                notice("path '%s' does not contain a '.devenv.flake.nix', searching up",path);
127:                    if (pathExists(path + "/.devenv.flake.nix")) {
131:                        throw Error("path '%s' is not part of a flake (neither it nor its parent directories contain a '.devenv.flake.nix' file)", path);
139:                    throw BadURL("could not find a .devenv.flake.nix file");
145:            if (!allowMissing && !pathExists(path + "/.devenv.flake.nix"))
146:                throw BadURL("path '%s' is not a flake (because it doesn't contain a '.devenv.flake.nix' file)", path);

src/libexpr/flake/call-flake.nix
73:          file = if builtins.pathExists (outPath + "/.devenv.flake.nix")
74:                 then "/.devenv.flake.nix"

src/libexpr/flake/flake.cc
181:    auto devEnvPath = rootDir / flakeDir / ".devenv.flake.nix";

Here are links to the relevant files on the relevant branch:

• libfetchers/git-utils.cc
• libexpr/flake/flakeref.cc
• libexpr/flake/call-flake.nix
• libexpr/flake/flake.cc

(It looks like the error message is one of the generic ones from libutil or libstore.)

I'm poking through those files now, but I'll soon have to switch gears for work so I may not be able to follow up today.

[domenkozar]

I suspect the issue is how Nix resolves paths, while .devenv.flake.nix is a red herring as it just looks it up and fallbacks to flake.nix

[domenkozar]

Testing a fix in #1137

[therealpxc]

As you've likely seen, this fix doesn't work because call-flake.nix still needs to look for devenv.nix at first, for print-dev-env to work in the devenv project.

Could there be a bug in builtins.pathExists that makes it fail for second-order dependencies when checking if .devenv.flake.nix exists? What else could this issue be?

Here's the relevant bit of my stack trace when I tested this fix:

❯ /nix/store/hdlb4zssxmmc5qa8926kqmp66pxxv757-devenv-1.0.4/bin/devenv shell
• Building shell ...
error:
       … while calling anonymous lambda
         at «nix-internal»/call-flake.nix:12:1:
           11| # unlocked trees.
           12| overrides:
             | ^
           13|

       … from call site
         at «nix-internal»/call-flake.nix:106:4:
          105|
          106| in allNodes.${lockFile.root}
             |    ^
          107|

       … while calling anonymous lambda
         at «nix-internal»/call-flake.nix:39:13:
           38|     builtins.mapAttrs
           39|       (key: node:
             |             ^
           40|         let

       error:
       error: path '/flake.nix' does not exist in Git repository '/Users/pxc/Documents/Code/devenv-problem-example'

✖ Command produced the following output:

✔ Building shell in 0.1s.
Error:   × Command `/nix/store/1zavp7qj3wic8b4vjnmsqbyg5dlpibn5-nix-devenv-2.21.0pre20240425_7c2a738/bin/nix --show-trace --extra-experimental-features nix-command --extra-experimental-
  │ features flakes --option warn-dirty false --option eval-cache false --keep-going --max-jobs 5 print-dev-env --profile /Users/pxc/Documents/Code/devenv-problem-example/.devenv/gc/
  │ shell` failed with with exit code 1

[domenkozar]

The bug is in pathExists and how it handles virtual paths. I'm trying to come up with a way to avoid doing the conditional in Nix code, but rather in C++.

[therealpxc]

Thanks for the update. I just noticed from a bit of playing around. You've probably seen it, but it might be helpful for posterity, or linking to the upstream Nix issue eventually, if it hasn't been filed yet:

devenv-problem-example on  main [!?⇡] 
❯ /nix/store/as7cgg7qvnkamrcycbgcdvhs0kv7wphx-nix-2.21-devenv/bin/nix eval --expr 'builtins.pathExists /nix/store/xvyy5vh6cg7958a26p2bqyz6jg5wkz4g-source/flake.nix'
false

devenv-problem-example on  main [!?⇡] 
❯ /nix/store/as7cgg7qvnkamrcycbgcdvhs0kv7wphx-nix-2.21-devenv/bin/nix eval --impure --expr 'builtins.pathExists /nix/store/xvyy5vh6cg7958a26p2bqyz6jg5wkz4g-source/flake.nix'
true

devenv-problem-example on  main [!?⇡] 
❯ /nix/store/as7cgg7qvnkamrcycbgcdvhs0kv7wphx-nix-2.21-devenv/bin/nix repl
Nix 2.21.0
Type :? for help.
nix-repl> builtins.pathExists /nix/store/xvyy5vh6cg7958a26p2bqyz6jg5wkz4g-source/flake.nix
true

pathExists seems to behave differently in the repl from outside the repl (perhaps because the repl is always implicitly an impure environment?), and also when --impure is passed vs. not. Perhaps even when flake evaluation is called with --impure, the impurity applies only to the top-level flake?

(I'm also a bit surprised that any reference to something inside the Nix store is forbidden even during 'pure' evaluation, and that pathExists simply returns false in that case rather than throwing an error. But I guess that's expected.)

[domenkozar]

This is mostly a side effect of NixOS/nix#6530, which tries to abstract away flake paths.

It should be reported there if we want it to be fixed upstream.

[domenkozar]

See also NixOS/nix#10505

[therealpxc]

Any word on this? This is a pretty killer bug for me. Should I just revert to older devenv for the time being?

[domenkozar]

This is on the top of my priority list of next week.

[npfp]

Hello, we got this error in our GitHub Actions (following https://devenv.sh/integrations/github-actions/#run-multiple-commands) since yesterday. Would you know how to revert to the previous version in the GitHub Actions?

[therealpxc]

Looks like this builtins.pathExists bug only affects flake inputs of type path, or at least it doesn't affect flake inputs of type github. By removing flake references of type path (paths that, incidentally, are already in /nix/store), I got my example repo to stop exhibiting the issue.

The repo my stuff was choking on was flake-parts. When I added it to my inputs and got an appropriate change in my lock file, this issue went away.

[LorenzBischof]

I have the same issue with git+ssh:// inputs. Adding flake: false works.

[tomaskallup]

I have a similar issue, but mine seems unrelated? As I get a bit different message and using flake: false doesn't help.

#1514