nix 2.6.0 transitive follows results in an error #6013

dvzubarev · 2022-01-29T17:49:43Z

Describe the bug
When using a chain of transitive follows an error is occurred when creating lock file.

Steps To Reproduce
Consider this chain of flakes z -> a -> b -> c.
The goal is to use the same nixpkgs in all packages.

git clone https://github.com/dvzubarev/flake-follows-bug-poc2.git
cd flake-follows-bug-poc2/z
nix flake update .

error: input 'a/b/c/nixpkgs' follows a non-existent input 'b/nixpkgs'
(use '--show-trace' to show detailed location information)

Expected behavior
I expect that nixpkgs that was set in z package will be propagated to all its inputs.
It works for a package.
E.g.

cd flake-follows-bug-poc2/a
nix flake update .
nix flake metadata .

warning: Git tree '/tmp/temp/flake-follows-bug-poc2' is dirty
Resolved URL:  git+file:///tmp/temp/flake-follows-bug-poc2?dir=a
Locked URL:    git+file:///tmp/temp/flake-follows-bug-poc2?dir=a
Path:          /nix/store/768kr4kadd8ddkda0wva35cz9h0j9hrl-source
Last modified: 2022-01-29 20:37:22
Inputs:
├───b: github:dvzubarev/flake-follows-bug-poc2/91cf8b9a7fcbd349ac54961b0d580d0c98c2d96d?dir=b
│   ├───c: github:dvzubarev/flake-follows-bug-poc2/46c67f57ff38672f0fe30e7af33fc5395c5e0730?dir=c
│   │   └───nixpkgs follows input 'b/nixpkgs'
│   └───nixpkgs follows input 'nixpkgs'
└───nixpkgs: github:nixos/nixpkgs/5bb20f9dc70e9ee16e21cc404b6508654931ce41

nix-env --version output
nix-env (Nix) 2.6.0

The text was updated successfully, but these errors were encountered:

amesgen · 2022-01-30T16:24:13Z

Many flake-based haskell.nix projects are affected by this (see https://input-output-hk.github.io/haskell.nix/tutorials/getting-started-flakes.html#scaffolding).

Bisecting points to 2664a21 / #5839 introducing this bug.

lovesegfault · 2022-01-31T21:30:26Z

cc. @balsoft

lilyball · 2022-02-02T00:17:47Z

I just ran into a similar issue in Nix 2.5.1 today where my equivalent to c's nixpkgs ended up being looked up in the registry (in this case it just referenced nixpkgs without declaring a URL for it). I can reproduce this issue with the POC here too, although for the POC c's nixpkgs ends up using the URL it was originally declared with.

> cd flake-follows-bug-poc2/z
> nix flake metadata
warning: creating lock file '/Users/lily/Dev/Scratch/flake-follows-bug-poc2/z/flake.lock'
warning: Git tree '/Users/lily/Dev/Scratch/flake-follows-bug-poc2' is dirty
Resolved URL:  git+file:///Users/lily/Dev/Scratch/flake-follows-bug-poc2?dir=z
Locked URL:    git+file:///Users/lily/Dev/Scratch/flake-follows-bug-poc2?dir=z
Path:          /nix/store/sh38niscjfvsjmn1hqh1jb3w65jkqcgk-source
Last modified: 2022-01-29 09:37:22
Inputs:
├───a: github:dvzubarev/flake-follows-bug-poc2/91cf8b9a7fcbd349ac54961b0d580d0c98c2d96d?dir=a
│   ├───b: github:dvzubarev/flake-follows-bug-poc2/46c67f57ff38672f0fe30e7af33fc5395c5e0730?dir=b
│   │   ├───c: github:dvzubarev/flake-follows-bug-poc2/46c67f57ff38672f0fe30e7af33fc5395c5e0730?dir=c
│   │   │   └───nixpkgs: github:nixos/nixpkgs/27249995942ba1718f7f05075d44c93bb8e0267c
│   │   └───nixpkgs follows input 'a/nixpkgs'
│   └───nixpkgs follows input 'nixpkgs'
└───nixpkgs: github:nixos/nixpkgs/efeefb2af1469a5d1f0ae7ca8f0dfd9bb87d5cfb

The a package also breaks, which is interesting. It declares that input b/c/nixpkgs changed from following b/nixpkgs to github:nixos/nixpkgs/27249995942ba1718f7f05075d44c93bb8e0267c. It looks like the checked-in a/flake.lock said that b/c/nixpkgs followed nixpkgs, but Nix 2.5.1 wants to update it to be broken:

> cd flake-follows-bug-poc2/a
> nix flake metadata
warning: updating lock file '/Users/lily/Dev/Scratch/flake-follows-bug-poc2/a/flake.lock':
• Updated input 'b/c/nixpkgs':
    follows 'b/nixpkgs'
  → 'github:nixos/nixpkgs/27249995942ba1718f7f05075d44c93bb8e0267c' (2022-02-01)
warning: Git tree '/Users/lily/Dev/Scratch/flake-follows-bug-poc2' is dirty
Resolved URL:  git+file:///Users/lily/Dev/Scratch/flake-follows-bug-poc2?dir=a
Locked URL:    git+file:///Users/lily/Dev/Scratch/flake-follows-bug-poc2?dir=a
Path:          /nix/store/bfkm2bdf6zhmj6pyk8n2qq2s1aqn6b0j-source
Last modified: 2022-01-29 09:37:22
Inputs:
├───b: github:dvzubarev/flake-follows-bug-poc2/46c67f57ff38672f0fe30e7af33fc5395c5e0730?dir=b
│   ├───c: github:dvzubarev/flake-follows-bug-poc2/46c67f57ff38672f0fe30e7af33fc5395c5e0730?dir=c
│   │   └───nixpkgs: github:nixos/nixpkgs/27249995942ba1718f7f05075d44c93bb8e0267c
│   └───nixpkgs follows input 'nixpkgs'
└───nixpkgs: github:nixos/nixpkgs/5bb20f9dc70e9ee16e21cc404b6508654931ce41

Even more bizarrely, after bouncing between the directories and checking nix flake metadata, I ran it in z again and it decided to update the dependencies again such that a/b/c/nixpkgs follows a/b/nixpkgs but a/b/nixpkgs is now pointing at a rev:

> cd flake-follows-bug-poc2/a
> nix flake metadata
warning: Git tree '/Users/lily/Dev/Scratch/flake-follows-bug-poc2' is dirty
warning: updating lock file '/Users/lily/Dev/Scratch/flake-follows-bug-poc2/z/flake.lock':
• Updated input 'a/b/c/nixpkgs':
    'github:nixos/nixpkgs/27249995942ba1718f7f05075d44c93bb8e0267c' (2022-02-01)
  → follows 'a/b/nixpkgs'
• Updated input 'a/b/nixpkgs':
    follows 'a/nixpkgs'
  → 'github:nixos/nixpkgs/efeefb2af1469a5d1f0ae7ca8f0dfd9bb87d5cfb' (2022-01-30)
warning: Git tree '/Users/lily/Dev/Scratch/flake-follows-bug-poc2' is dirty
Resolved URL:  git+file:///Users/lily/Dev/Scratch/flake-follows-bug-poc2?dir=z
Locked URL:    git+file:///Users/lily/Dev/Scratch/flake-follows-bug-poc2?dir=z
Path:          /nix/store/ysby4xpi2ljg9721xfrvx21jz6d4i7sc-source
Last modified: 2022-01-29 09:37:22
Inputs:
├───a: github:dvzubarev/flake-follows-bug-poc2/91cf8b9a7fcbd349ac54961b0d580d0c98c2d96d?dir=a
│   ├───b: github:dvzubarev/flake-follows-bug-poc2/46c67f57ff38672f0fe30e7af33fc5395c5e0730?dir=b
│   │   ├───c: github:dvzubarev/flake-follows-bug-poc2/46c67f57ff38672f0fe30e7af33fc5395c5e0730?dir=c
│   │   │   └───nixpkgs follows input 'a/b/nixpkgs'
│   │   └───nixpkgs: github:nixos/nixpkgs/efeefb2af1469a5d1f0ae7ca8f0dfd9bb87d5cfb
│   └───nixpkgs follows input 'nixpkgs'
└───nixpkgs: github:nixos/nixpkgs/efeefb2af1469a5d1f0ae7ca8f0dfd9bb87d5cfb

Running it again updates /a/b/c/nixpkgs once more to stop following, which produces a metadata tree with 3 different versions of nixpkgs now:

> nix flake metadata
warning: Git tree '/Users/lily/Dev/Scratch/flake-follows-bug-poc2' is dirty
warning: updating lock file '/Users/lily/Dev/Scratch/flake-follows-bug-poc2/z/flake.lock':
• Updated input 'a/b/c/nixpkgs':
    follows 'a/b/nixpkgs'
  → 'github:nixos/nixpkgs/27249995942ba1718f7f05075d44c93bb8e0267c' (2022-02-01)
warning: Git tree '/Users/lily/Dev/Scratch/flake-follows-bug-poc2' is dirty
Resolved URL:  git+file:///Users/lily/Dev/Scratch/flake-follows-bug-poc2?dir=z
Locked URL:    git+file:///Users/lily/Dev/Scratch/flake-follows-bug-poc2?dir=z
Path:          /nix/store/bx5z8qhq8scivyjb4nb2z80rghd3623d-source
Last modified: 2022-01-29 09:37:22
Inputs:
├───a: github:dvzubarev/flake-follows-bug-poc2/91cf8b9a7fcbd349ac54961b0d580d0c98c2d96d?dir=a
│   ├───b: github:dvzubarev/flake-follows-bug-poc2/46c67f57ff38672f0fe30e7af33fc5395c5e0730?dir=b
│   │   ├───c: github:dvzubarev/flake-follows-bug-poc2/46c67f57ff38672f0fe30e7af33fc5395c5e0730?dir=c
│   │   │   └───nixpkgs: github:nixos/nixpkgs/27249995942ba1718f7f05075d44c93bb8e0267c
│   │   └───nixpkgs: github:nixos/nixpkgs/efeefb2af1469a5d1f0ae7ca8f0dfd9bb87d5cfb
│   └───nixpkgs follows input 'nixpkgs'
└───nixpkgs: github:nixos/nixpkgs/efeefb2af1469a5d1f0ae7ca8f0dfd9bb87d5cfb

This is actually reproducible. If I reset the repo, running nix flake metadata within z three times in a row produces the same sequence of modifications.

My guess here is that the issue with Nix 2.6.0 is related to what I'm describing, perhaps it was a partial fix to the issue I'm showing but that partial fix just results in different broken behavior.

lovesegfault · 2022-02-02T00:22:46Z

@lilyball I think you're seeing this issue: #5728?

That patch is included in 2.6.0

lilyball · 2022-02-02T00:44:35Z

@lovesegfault Interesting, I didn't realize nix flake update would produce a different lockfile than is produced by nix flake metadata. I do see that running nix flake update wants to make changes, though it still produces an incorrect resulting graph. If I'm reading this lockfile correctly (since I can't ask nix flake metadata to print it for me), nix flake update from z produces something that looks like

Inputs:
├───a: github:dvzubarev/flake-follows-bug-poc2/91cf8b9a7fcbd349ac54961b0d580d0c98c2d96d?dir=a
│   ├───b: github:dvzubarev/flake-follows-bug-poc2/46c67f57ff38672f0fe30e7af33fc5395c5e0730?dir=b
│   │   ├───c: github:dvzubarev/flake-follows-bug-poc2/46c67f57ff38672f0fe30e7af33fc5395c5e0730?dir=c
│   │   │   └───nixpkgs: github:nixos/nixpkgs/fd8529c6a2e3bde16999618eede09ee0afe18d0f
│   │   └───nixpkgs: follows input 'a/nixpkgs'
│   └───nixpkgs: github:nixos/nixpkgs/fd8529c6a2e3bde16999618eede09ee0afe18d0f
└───nixpkgs: github:nixos/nixpkgs/efeefb2af1469a5d1f0ae7ca8f0dfd9bb87d5cfb

So this is still wrong, though different than what nix flake metadata does. I haven't tried Nix 2.6.0 as it's not in nixpkgs-unstable so I'm curious what the exact behavioral difference is.

lilyball · 2022-02-02T00:48:37Z

Ok I just tested Nix 2.6.0, turns out it never even creates a lockfile for z. Oops.

lilyball · 2022-02-02T01:02:31Z

Poking at Nix 2.6.0 vs Nix 2.5.1 some more and I think you're right, the behavior I'm describing is unrelated. And the fact that Nix 2.5.1's nix flake update produces the wrong structure is probably unrelated as well. I suspect the issue with the bisected commit has to do with the bit where it changes an "absolute" flag from false to true. I don't know this code, but that sure sounds like it's related to a/b/nixpkgs being turned into b/nixpkgs.

balsoft · 2022-02-02T07:35:21Z

I'll take a look

It's not possible in general to know in computeLocks, relative to which path the follows was intended to be. So, we always resolve follows to their absolute states when we encounter them (which can either be in parseFlakeInput or computeLocks' fake input population). Fixes NixOS#6013 Fixes NixOS#5609 Fixes NixOS#5697 (again)

balsoft · 2022-02-02T18:54:03Z

Resolved in #6036, please test/review

lilyball · 2022-02-02T21:52:18Z

@balsoft That PR does make this POC work, though the resulting lockfile for z ends up recording

    "c": {
      "inputs": {
        "nixpkgs": [
          "a",
          "b",
          "nixpkgs"
        ]
      },

which is curious as it should be able to record "nixpkgs": [ "b", "nixpkgs" ] (as looking that up in the "b" node then resolves to a/nixpkgs).

lilyball · 2022-02-02T21:53:37Z

This doesn't seem harmful, just unexpected, as it means changing b.inputs.nixpkgs.follows will cause the entry for c to change. I suppose it does make it easier to read the lockfile and figure out what the path from the root is though.

balsoft · 2022-02-02T22:01:21Z

FTR the behavior on 2.4 was the same:

$ nix --version
nix (Nix) 2.4
$ rm flake.lock
$ nix flake update
warning: creating lock file '/home/balsoft/projects/flake-follows-bug-poc2/z/flake.lock'
warning: Git tree '/home/balsoft/projects/flake-follows-bug-poc2' is dirty
$ cat flake.lock | jq .nodes.c.inputs.nixpkgs
[
  "a",
  "b",
  "nixpkgs"
]

Since follows in the lockfile are supposed to be relative to the flake root (=absolute).

NixOS/nix#6013

dvzubarev added the bug label Jan 29, 2022

edolstra added flakes regression Something doesn't work anymore labels Feb 1, 2022

edolstra added this to the nix-2.7 milestone Feb 1, 2022

balsoft mentioned this issue Feb 2, 2022

Flake follows: resolve all follows to absolute #6036

Merged

edolstra closed this as completed in #6036 Feb 2, 2022

tomberek mentioned this issue Feb 17, 2022

Follow up message when creating a new project #4503

Closed

edrex mentioned this issue Feb 27, 2022

flakes: failing follows test case #6171

Open

amesgen mentioned this issue Mar 1, 2022

update pins input-output-hk/haskell.nix#1371

Merged

the-mikedavis mentioned this issue Mar 9, 2022

The nix flake is broken after pull request #1758 helix-editor/helix#1779

Closed

Infinidoge added a commit to Infinidoge/universe that referenced this issue Mar 30, 2023

fix(flake): bud beautysh inputs

94a51da

NixOS/nix#6013

Infinidoge added a commit to Infinidoge/universe that referenced this issue Mar 30, 2023

fix(flake): bud beautysh inputs

6a02e7b

NixOS/nix#6013

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nix 2.6.0 transitive follows results in an error #6013

nix 2.6.0 transitive follows results in an error #6013

dvzubarev commented Jan 29, 2022 •

edited

Loading

amesgen commented Jan 30, 2022

lovesegfault commented Jan 31, 2022

lilyball commented Feb 2, 2022

lovesegfault commented Feb 2, 2022

lilyball commented Feb 2, 2022

lilyball commented Feb 2, 2022

lilyball commented Feb 2, 2022

balsoft commented Feb 2, 2022

balsoft commented Feb 2, 2022

lilyball commented Feb 2, 2022

lilyball commented Feb 2, 2022

balsoft commented Feb 2, 2022

nix 2.6.0 transitive follows results in an error #6013

nix 2.6.0 transitive follows results in an error #6013

Comments

dvzubarev commented Jan 29, 2022 • edited Loading

amesgen commented Jan 30, 2022

lovesegfault commented Jan 31, 2022

lilyball commented Feb 2, 2022

lovesegfault commented Feb 2, 2022

lilyball commented Feb 2, 2022

lilyball commented Feb 2, 2022

lilyball commented Feb 2, 2022

balsoft commented Feb 2, 2022

balsoft commented Feb 2, 2022

lilyball commented Feb 2, 2022

lilyball commented Feb 2, 2022

balsoft commented Feb 2, 2022

dvzubarev commented Jan 29, 2022 •

edited

Loading