flakes: nix shell (and similar) fails, if flakes.nix inside a git-repo ist no yet added to the repo (error: getting status of '/nix/store/.../flake.nix': No such file or directory)

[systemofapwne]

If a flakes.nix file is copied to a repository (empty or with content), then without adding it yet to the reposiroty, envoking nix shell and other flake related commands will fail with an error like

error: getting status of '/nix/store/0ccnxa25whszw7mgbgyzdm4nqc0zwnm8-source/flake.nix': No such file or directory

rather than falling back to just use the locally available flake.nix (should be the default IMH).

Steps To Reproduce

1. git init test && cd test
2. Copy any flake.nix file in there (see e.g. my attached one)
3. run nix shell
4. See error

Expected behavior

A nix shell should be opened using the locally present but not yet added or commited flakes.nix.

nix-env --version output
nix-env (Nix) 2.8.1

Additional context

The flake.nix file, that I used

{
  inputs.artiq.url = "git+https://github.com/m-labs/artiq.git";
  inputs.extrapkg.url = "git+https://git.m-labs.hk/M-Labs/artiq-extrapkg.git";
  inputs.extrapkg.inputs.artiq.follows = "artiq";
  outputs = { self, artiq, extrapkg }:
    let
      pkgs = artiq.inputs.nixpkgs.legacyPackages.x86_64-linux;
      aqmain = artiq.packages.x86_64-linux;
      aqextra = extrapkg.packages.x86_64-linux;
    in {
      defaultPackage.x86_64-linux = pkgs.buildEnv {
        name = "artiq-env";
        paths = [
          # ========================================
          # EDIT BELOW
          # ========================================
          (pkgs.python3.withPackages(ps: [
            # List desired Python packages here.
            aqmain.artiq
            #ps.paramiko  # needed if and only if flashing boards remotely (artiq_flash -H)
            #aqextra.flake8-artiq

            # The NixOS package collection contains many other packages that you may find
            # interesting. Here are some examples:
            ps.pandas
            ps.numpy
            ps.scipy
            #ps.numba
            #ps.matplotlib
            # or if you need Qt (will recompile):
            #(ps.matplotlib.override { enableQt = true; })
            ps.matplotlib.override { enableGtk3 = true; }
            ps.pybase64
            #ps.bokeh
            #ps.cirq
            #ps.qiskit
          ]))
          #aqextra.korad_ka3005p
          #aqextra.novatech409b
          # List desired non-Python packages here
          #aqmain.openocd-bscanspi  # needed if and only if flashing boards
          # Other potentially interesting packages from the NixOS package collection:
          #pkgs.gtkwave
          #pkgs.spyder
          #pkgs.R
          #pkgs.julia
          # ========================================
          # EDIT ABOVE
          # ========================================
        ];
      };
    };
}

[aakropotkin]

This is the expected behavior AFAIK

[tomberek]

Summary: I'm not aware of a simple and easy solution to this UX annoyance that does not lead to more UX problems elsewhere.

This is intended. While making this new behavior default would simplify a few situations, it will also make others more complicated. As an analogy to another technology that purports to provide reproducible source code trees (git) that does not provide a similar default behavior:

1. git init test && cd test
2. copy any file in there
3. run git show
   See error

Perhaps the error message should be better, or there should be an easier way than git --intent-to-add to do what you'd like. Perhaps it should automatically include it, but with a warning that clarifies what is happening, but then we must be careful to avoid this becoming depended on and then authors accidentally leave files out of repos that are needed for evaluation to occur. Then all this behavior needs to be explained. For now, the rule is simple: "if you are in a git repo, flake evaluation only utilizes files that git is tracking."

[aakropotkin]

Honestly the fact that flakes don't automatically add files is a feature, and for me it's one of the biggest draws because of its interaction with the eval cache.

Sure sometimes I forget to add a file, but I almost always forget to gitignore things like result in a fresh repo. If files were auto added without a flag or option, the eval cache would trigger rebuilds as often as nix-build used to.

[systemofapwne]

Summary: I'm not aware of a simple and easy solution to this UX annoyance that does not lead to more UX problems elsewhere.

This is intended. While making this new behavior default would simplify a few situations, it will also make others more complicated. As an analogy to another technology that purports to provide reproducible source code trees (git) that does not provide a similar default behavior:

1. git init test && cd test
2. copy any file in there
3. run git show
   See error

I can understand, that this is by design and intended. I can even live with that decision. Regarding your next statement...

Perhaps the error message should be better, or there should be an easier way than git --intent-to-add to do what you'd like.

I am absolutely for that. I spent about two hours debugging this and I am and will not be the last one falling for that. The error message should be way more clearer for a better UX.
Like

error: getting status of '/nix/store/0ccnxa25whszw7mgbgyzdm4nqc0zwnm8-source/flake.nix': No such file or directory
Local flake.nix found, which is not member of the repository. Consider adding it first.

or something similar.

[systemofapwne]

Honestly the fact that flakes don't automatically add files is a feature, and for me it's one of the biggest draws because of its interaction with the eval cache.

Sure sometimes I forget to add a file, but I almost always forget to gitignore things like result in a fresh repo. If files were auto added without a flag or option, the eval cache would trigger rebuilds as often as nix-build used to.

Maybe (or certainly) I do not understand the full nix architecture in detail yet.
But I was rather referring to making flakes not fail when a flakes.nix file certainly is there (but not a member of a repo yet). Not automatically adding it to the repo.
Yet, as stated one post earlier, I can live with that behavior/design choice. The error message should just be more clearer, so that someone without a deep understanding of nix, flakes etc can troubleshoot the situation with a simple git add flakes.nix.

[yeoldegrove]

I have the very same issue. My use case is this https://nixos.wiki/wiki/Flakes#Super_fast_nix-shell
I am using this for repos that might also be used by non-nix users that would get annoyed by adding shell.nix and flake.* files to the repos.
In the old world, I had my shell.nix in .gitignore (no flake.nix) and everything worked fine.
Also adding the flake.nix file to .gitignore will not be enough to get rid of "flake.nix': No such file or directory".

[aakropotkin]

I have the very same issue. My use case is this https://nixos.wiki/wiki/Flakes#Super_fast_nix-shell
I am using this for repos that might also be used by non-nix users that would get annoyed by adding shell.nix and flake.* files to the repos.
In the old world, I had my shell.nix in .gitignore (no flake.nix) and everything worked fine.
Also adding the flake.nix file to .gitignore will not be enough to get rid of "flake.nix': No such file or directory".

Ignoring flake.nix would prevent your flake from ever being available. Definitely don't add that to your ignores.

I think you misunderstood my comment about how legacy UI commands would cause spurious rebuilds because unimportant files were added to the working dir. The common workaround was various source filters, notable nix-gitignore. The old workflow was basically "clone a repo, extend .gitignore ( to the point that it's almost identical to git clean -xfd ), create *.nix using nix-gitignore filters, and THEN run nix-*.

Skipping these steps would cause Nix to rebuild projects needlessly, often to such a degree that it was easier for Nix files to be maintained externally. The new UI'S git integration resolves these issues. The tradeoff is "I don't have to do all that filtering, I just have to git add flake.{nix,lock} once and forget about it". 😉

[tomberek]

I am absolutely for that. I spent about two hours debugging this and I am and will not be the last one falling for that. The error message should be way more clearer for a better UX.

There is a pending PR which has a better message. What do you think about this?

nix/src/libfetchers/git.cc

Lines 756 to 759 in 5f13402

	if (nix::pathExists(path.abs()))
	return RestrictedPathError("access to path '%s' is forbidden because it is not under Git control; maybe you should 'git add' it to the repository '%s'?", path, url);
	else
	return RestrictedPathError("path '%s' does not exist in Git repository '%s'", path, url);

[systemofapwne]

I am absolutely for that. I spent about two hours debugging this and I am and will not be the last one falling for that. The error message should be way more clearer for a better UX.

There is a pending PR which has a better message. What do you think about this?

nix/src/libfetchers/git.cc

Lines 756 to 759 in 5f13402

	if (nix::pathExists(path.abs()))
	return RestrictedPathError("access to path '%s' is forbidden because it is not under Git control; maybe you should 'git add' it to the repository '%s'?", path, url);
	else
	return RestrictedPathError("path '%s' does not exist in Git repository '%s'", path, url);

This is a welcome solution for me!

[WhatisRT]

I've also lost several hours to this issue. Since it wasn't completely non-trivial to find, the PR that contains this change is #6530. That PR is also still a draft, so I'd prefer this to stay open until it's merged.

[lheckemann]

@tomberek / @edolstra could one of you reopen this issue and put a "Closes #6642" in the PR description for #6530?

[yacinehmito]

Like @WhatisRT, I lost hours to this issue. @tomberek Please follow @WhatisRT's suggestion and reopen the issue. Thank you.

[tejing1]

This situation might be improved by making the nix CLI default to path:. instead of git+file:. when flake.nix is not added to git and/or is gitignored.

The usecase of adding a local flake.nix to a repo which would not accept it being committed upstream would be streamlined this way, though it wouldn't stamp out the unintuitive issues related to files not existing due to being untracked or ignored by git. It might even add to them, since adding flake.nix would change the behavior of other files in this regard.

[zippy]

That error message is indeed terrible UX, it provides no clue as to what's going on.

[aakropotkin]

This situation might be improved by making the nix CLI default to path:. instead of git+file:. when flake.nix is not added to git and/or is gitignored.

The usecase of adding a local flake.nix to a repo which would not accept it being committed upstream would be streamlined this way, though it wouldn't stamp out the unintuitive issues related to files not existing due to being untracked or ignored by git. It might even add to them, since adding flake.nix would change the behavior of other files in this regard.

Now this I actually agree with.

For clarity though you should add flake{nix, lock} to your repo. Yes the error message could be better ( PR it ).

[steveej]

This situation might be improved by making the nix CLI default to path:. instead of git+file:. when flake.nix is not added to git and/or is gitignored.

in many situations that would also lead to copying lots of data into the nix store, which i would find to be a regression. people would have to readjust their habits to explicitly state git+file. which i certainly feel resistance towards (i.e. laziness 😆).

since in the case of a dirty git tree users already get a warning, i suggest to also print a warning if flake.{nix,lock} exist but are not racked in git.

[tejing1]

in many situations that would also lead to copying lots of data into the nix store, which i would find to be a regression. people would have to readjust their habits to explicitly state git+file. which i certainly feel resistance towards (i.e. laziness ).

It would only cause that in situations where currently it fails completely. When flake.nix is known to git, it would still default to git+file:., just as it does now.

Moving from non-functionality to poor performance doesn't seem like a regression to me.

[steveej]

It would only cause that in situations where currently it fails completely. When flake.nix is known to git, it would still default to git+file:., just as it does now.

ah, i interpreted your suggestion as changing the default unconditionally. but you meant it to be dependent on the flake.{nix,lock} tracking status, did i get it right now? if so it'd deem that acceptable, however still somewhat of a foot-gun. how about in that case nix just acts as if flake.{nix,lock} where tracked?

[tejing1]

Yes, that's what I meant.

Indeed, it might be better to just pretend flake.{nix,lock} are tracked even when they aren't. I haven't thought it all through, but it does seem like that has less potential fallout, and avoids copying .git to the store all the time, too.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/doing-the-zero-to-nix-and-get-a-random-error/27692/8

[jerlam06]

2 years since the issue creation and we still cannot run nix develop when the flake.* files are in gitignore? This is my company's app and do not want to commit these flake files which are not related to the project and which are only used by me. Just let me do what I want instead of imposing me this decision...

[danielpza]

@jerlam06 have you tried nix develop path:.?

[tejing1]

The problem is that there isn't really a good solution here.

If nix copies non-git files generally, then you end up with all kinds of detritus and intermediate files copied into the nix store, and then those intermediate files often mess up build processes because the timestamp data is gone.

If you build only from the latest actual commit then it's too annoying to quickly test out a change to a flake.

The compromise we've landed on is to use the dirty copies of any files tracked by git, but ignore other files. It's not perfect, but it's quite hard to come up with a change to it that wouldn't cost too much in terms of other unintuitive results. I'd love to see a nice solution to the usecase of adding a local flake.nix to a repo that doesn't have one upstream, but any such solution needs be considered in terms of all the effects it would have, not just that it solves that one issue.

[mightybyte]

I don't think we should be forced to overload .gitignore to serve this separate purpose of defining what goes into the nix store. Maybe have that as a default, but it needs to be overridable. I just tried using nix flakes for a new project for the first time and this is the very first use case I wanted. I think using flake.nix to nixify a project without having to check the flake.nix file into the repo is a very important use case.

[mwoodpatrick]

I hit the same issue and was very confused by this, Fixing the error message as per the above would have really helped me and would be a very welcome change. This is an issue that should be well documented in flakes documentation. But this is the first time I have seen this behavior documented.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/problem-running-nix-develop/52249/3

[zeratax]

I hit the same issue and was very confused by this, Fixing the error message as per the above would have really helped me and would be a very welcome change. This is an issue that should be well documented in flakes documentation. But this is the first time I have seen this behavior documented.

this seems to have been fixed in #9398 but sadly isnt in the nix that ships with 24.05 :/

[systemofapwne]

I hit the same issue and was very confused by this, Fixing the error message as per the above would have really helped me and would be a very welcome change. This is an issue that should be well documented in flakes documentation. But this is the first time I have seen this behavior documented.

this seems to have been fixed in #9398 but sadly isnt in the nix that ships with 24.05 :/

This looks like a regression caused by this PR: kolloch@598deb2

Since then the whole flake system also was under heavy development and refactoring.