'nix profile' has terrible experience upgrading packages

[domenkozar]

Here's a user session from devenv discord:

❯ nix-env -if https://github.com/cachix/devenv/tarball/v0.5
error: profile '/nix/var/nix/profiles/per-user/i97henka/profile' is incompatible with 'nix-env'; please use 'nix profile' instead

❯ nix profile install github:cachix/devenv/v0.5 --extra-experimental-features flakes
warning: Using saved setting for 'extra-substituters = https://devenv.cachix.org' from ~/.local/share/nix/trusted-settings.json.
warning: Using saved setting for 'extra-trusted-public-keys = devenv.cachix.org-1:w1cLUi8dv3hnoSPGAuibQv+f9TZLr6cv/Hm9XgU50cw=' from ~/.local/share/nix/trusted-settings.json.
error: files '/nix/store/k0gwhiwfs8c218kph78vx75bph51iw31-devenv/bin/devenv' and '/nix/store/7i4hxhw44lr7r64zlmp4zq7cqpsx7smw-devenv/bin/devenv' have the same priority 5; use 'nix-env --set-flag priority NUMBER INSTALLED_PKGNAME' or type 'nix profile install --help' if using 'nix profile' to find out howto change the priority of one of the conflicting packages (0 being the highest priority)

Any pointers on how I can get the upgrade working would be much appreciated!

the fix

nix profile list and nix profile remove /nix/store/xxxxx

[matthewbauer]

I've had similar experiences with nix profile.

A couple other related issues:

• nix profile upgrade doesn't seem to match on the flake url correctly:

$ nix profile install github:cachix/devenv/v0.5#defaultPackage.aarch64-darwin
$ nix profile upgrade github:cachix/devenv/v0.5#defaultPackage.aarch64-darwin
warning: 'github:cachix/devenv/v0.5#defaultPackage.aarch64-darwin' does not match any packages
warning: Use 'nix profile list' to see the current profile.
$ nix profile upgrade '.*devenv.*'
warning: '.*devenv.*' does not match any packages
warning: Use 'nix profile list' to see the current profile.

while this does work:

$ nix profile upgrade defaultPackage.aarch64-darwin

• On Nix 2.12, I can create duplicate entries in my profile as long as the hashes are the same:

$ nix profile install github:cachix/devenv/v0.5#defaultPackage.aarch64-darwin
$ nix profile install github:cachix/devenv/v0.5#defaultPackage.aarch64-darwin
$ nix profile install github:cachix/devenv/v0.5#defaultPackage.aarch64-darwin
$ nix profile list
...
15 github:cachix/devenv/v0.5#defaultPackage.aarch64-darwin github:cachix/devenv/ba6818f4c39fd95aebdb6dc441401f2b60484652#defaultPackage.aarch64-darwin /nix/store/30llcvl84z9ns52hs0sz47sqrsakfdz6-devenv
16 github:cachix/devenv/v0.5#defaultPackage.aarch64-darwin github:cachix/devenv/ba6818f4c39fd95aebdb6dc441401f2b60484652#defaultPackage.aarch64-darwin /nix/store/30llcvl84z9ns52hs0sz47sqrsakfdz6-devenv
17 github:cachix/devenv/v0.5#defaultPackage.aarch64-darwin github:cachix/devenv/ba6818f4c39fd95aebdb6dc441401f2b60484652#defaultPackage.aarch64-darwin /nix/store/30llcvl84z9ns52hs0sz47sqrsakfdz6-devenv

These would end up conflicting when you upgrade I think.

[matthewbauer]

Haven't tested it , but maybe something like this would help for the first problem:

diff --git a/src/nix/profile.cc b/src/nix/profile.cc
index 11910523d..60393d87b 100644
--- a/src/nix/profile.cc
+++ b/src/nix/profile.cc
@@ -369,7 +369,7 @@ class MixProfileElementMatchers : virtual Args
                 if (element.storePaths.count(store.parseStorePath(*path))) return true;
             } else if (auto regex = std::get_if<RegexPattern>(&matcher)) {
                 if (element.source
-                    && std::regex_match(element.source->attrPath, regex->reg))
+                    && std::regex_match(element.source->originalRef.to_string() + "#" + element.source->attrPath + printOutputsSpec(element.source->outputs) : "-", regex->reg))
                     return true;
             }
         }

[domenkozar]

Reminds me of #4482

[bobvanderlinden]

Maybe nix profiles need to be rethought of. Both upgrade and remove have a bad experience, even if you know your way around Nix.

The problem is that Nix doesn't know about simple names of individual packages. For instance, installing the latest version of difftastic using:

nix profile install github:nixos/nixpkgs/nixos-unstable#difftastic

Will nicely install difftastic, but I cannot refer to the entry using difftastic, which is what I presume would be possible using nix profile remove difftastic.

Would it be possible to store (for each entry in profile) a simple name that is (by default) inferred from the "installable" argument?

$ nix profile install --entry difftastic github:nixos/nixpkgs/nixos-unstable#difftastic
$ nix profile remove difftastic

By default the 'entry-name' would be inferred from the URL. The name would be set by conditions in order upon installing:

1. The --entry flag
2. Anything behind #: for instance github:nixos/nixpkgs/nixos-unstable#difftastic -> difftastic
3. Any repository segment: for instance github:cachix/devenv/v0.5 -> devenv.
4. Error on not being able to infer the name and suggest to use --entry.

This would also in many cases avoid using the regular expression feature. This feature currently could be considered bad practice, as it quickly results in unintended errors. As well as bad UX, as it needs ' in shells and needing to know regex vs globbing.

If you then install something that has the same 'entry-name', the installation can warn or error that the entry name already exists and maybe asks or suggests how to upgrade the package to the new URL.

The numbers would also not be needed anymore. Those numbers seem like a kind of priority, but priority is something entirely different. The numbers are currently merely identifiers for installed packages: much better suited for unique names.

Would something like this be viable for profiles?

EDIT: I did some more digging. The manifest.json currently looks like:

{
  "elements": [
    {
      "active": true,
      "attrPath": "legacyPackages.x86_64-linux.difftastic",
      "originalUrl": "github:nixos/nixpkgs/nixos-unstable",
      "outputs": null,
      "priority": 5,
      "storePaths": [
        "/nix/store/87a50y83m4hg20ms5ch5x2vjdi7lzs0s-difftastic-0.43.0"
      ],
      "url": "github:nixos/nixpkgs/0591d6b57bfeb55dfeec99a671843337bc2c3323"
    },
    ...
  ],
  "version": 2
}

It could look like:

{
  "elements": {
    "difftastic": {
      "active": true,
      "attrPath": "legacyPackages.x86_64-linux.difftastic",
      "originalUrl": "github:nixos/nixpkgs/nixos-unstable",
      "outputs": null,
      "priority": 5,
      "storePaths": [
        "/nix/store/87a50y83m4hg20ms5ch5x2vjdi7lzs0s-difftastic-0.43.0"
      ],
      "url": "github:nixos/nixpkgs/0591d6b57bfeb55dfeec99a671843337bc2c3323"
    },
    ...
  },
  "version": 3
}

EDIT2: I don't want to go overboard in this issue, but in terms of UX it would be even nicer if this could integrate better with declarative configurations like home-manager as well. Make it possible to have manifest.json be part of your git-tracked home-manager configuration. Maybe refer to a manifest.json file from home-manager or maybe even see these 'nix profile elements' as flake inputs. To some extend, it could prevent users avoiding nix profile, because it doesn't integrate well with declarative (and shareable) configurations.

[bobvanderlinden]

There are quite a few UX problems with nix profile. I'd like to document them to give some context:

$ nix profile install github:cachix/devenv/latest
$ nix profile upgrade github:cachix/devenv/latest
warning: 'github:cachix/devenv/latest' does not match any packages
warning: Use 'nix profile list' to see the current profile.
$ nix profile remove devenv
warning: 'devenv' does not match any packages
warning: Use 'nix profile list' to see the current profile.

• The installable used in install is not applicable to remove nor upgrade.
• The output shows warnings, but are actually errors. Nothing has changed to the profile.
• nix profile list is suggested, but it isn't clear that you need to pick the index from that list.
• The name of the installable (devenv) is not applicable to remove nor upgrade. Ideally the NAME is deferred from the flake patterns (github:OWNER/NAME/VERSION, github:OWNER/REPO/VERSION#NAME, nixpkgs#NAME, github:OWNER/NAME/VERSION#default).

---

The main identifier for package entries is the index. The index is quite bad as an identifier:

$ nix profile list
....
3 flake:nixpkgs#legacyPackages.x86_64-linux.vim github:NixOS/nixpkgs/9952d6bc395f5841262b006fbace8dd7e143b634#legacyPackages.x86_64-linux.vim /nix/store/9hcby08vmd6k3zl3clwvlw5y5vkmc3z6-vim-9.0.0609
4 flake:nixpkgs#legacyPackages.x86_64-linux.cowsay github:NixOS/nixpkgs/9952d6bc395f5841262b006fbace8dd7e143b634#legacyPackages.x86_64-linux.cowsay /nix/store/azn0g0m6yg6m9vmdp3wq6wjbsd1znv44-cowsay-3.7.0 /nix/store/qgsl9bd6vvnpjaxhdnzg0xmycg3652c0-cowsay-3.7.0-man
5 flake:nixpkgs#legacyPackages.x86_64-linux.nano github:NixOS/nixpkgs/9952d6bc395f5841262b006fbace8dd7e143b634#legacyPackages.x86_64-linux.nano /nix/store/hhr8ijr2p5qf3lyklzyw8cndhsmpklsv-nano-7.2
$ nix profile remove 3
removing 'flake:nixpkgs#legacyPackages.x86_64-linux.vim'
$ nix profile remove 4
removing 'flake:nixpkgs#legacyPackages.x86_64-linux.nano'

In this case the numeric identifier in nix profile shifts with every change. In this case the user intended to remove vim and cowsay, but instead removed vim and nano. Very confusing.

• The identifier in nix profile should be stable.

---

As an alternative to the identifier, there is the use of regexp to match specific packages. This usually results in confusion:

Forgetting quotation in most shells:

$ nix profile remove .*vim.*
fish: No matches for wildcard '.*vim.*'. See `help wildcards-globbing`.
nix profile remove .*vim.*
                   ^~~~~~^

$ nix profile remove .*vim.*
zsh: no matches found: .*vim.*

Using wildcards instead of regexp result in confusing error message:

$ nix profile remove '*vim*'
error: Mismatched '(' and ')' in regular expression

Which ( and )?

Forgetting regexp and wildcards altogether:

$ nix profile remove vim
warning: 'vim' does not match any packages
warning: Use 'nix profile list' to see the current profile.

This makes it seem like vim was not installed at all, but it just wasn't matched by the regexp vim, because it should match flake:nixpkgs#legacyPackages.x86_64-linux.vim.

This imo is also solved when using proper identifiers for packages.

---

Multiple installs result in multiple entries, even though they conflict with each-other:

$ nix profile install github:cachix/devenv/latest
$ nix profile install github:cachix/devenv/latest
$ nix profile install github:cachix/devenv/latest
$ nix profile list
...
3 github:cachix/devenv/latest#defaultPackage.x86_64-linux github:cachix/devenv/15445416534c6253a78c4eb1f803af0338ead497#defaultPackage.x86_64-linux /nix/store/j5v7sh1ii9vv1bw9ngzdh5h76hh0zljh-devenv
4 github:cachix/devenv/latest#defaultPackage.x86_64-linux github:cachix/devenv/15445416534c6253a78c4eb1f803af0338ead497#defaultPackage.x86_64-linux /nix/store/j5v7sh1ii9vv1bw9ngzdh5h76hh0zljh-devenv
5 github:cachix/devenv/latest#defaultPackage.x86_64-linux github:cachix/devenv/15445416534c6253a78c4eb1f803af0338ead497#defaultPackage.x86_64-linux /nix/store/j5v7sh1ii9vv1bw9ngzdh5h76hh0zljh-devenv

• nix install should error when the same package is attempted to be installed. Preferably explaining about nix profile upgrade and possibly nix profile remove

---

Installing packages from different sources results in huge downloads. It's hard to avoid this:

$ nix profile install github:nixos/nixpkgs/nixpkgs-unstable#cowsay
# A week later:
$ nix profile install github:nixos/nixpkgs/nixpkgs-unstable#vim

Now 2 different versions of nixpkgs are pinned this has a few downsides:

• It results in huge downloads (long waiting time)
• It results in huge disk space usage (until all packages are upgraded and store GCed)
• It behaves differently from most package managers, where the package manager is usually pinned to a single package set until you explicitly tell it to update the package set (apt update, pacman -y, etc).
• It can result in incompatibilities when plugin-like libraries (like in lib/gtk-3.0, lib/qt4/plugins) are used across multiple packages from differently pinned nixpkgs. These libraries are linked against different versions of glibc, which can result in runtime problems. It's fine if the user explicitly wants to do this, but currently it's the default with nix profile.

There are some advantages to having a nixpkgs-pin per package, but currently it is the default and shouldn't be something the user needs to handle.

---

The main selling point of Nix is reproducibility. However, since nix profile doesn't expose its manifest.json it is far from trivial to reproduce the same profile on a different system. manifest.json seems like a good candidate to be used in nix shell, home-manager and NixOS. This gives the best of both worlds: imperative CLI, declarative and shareable configuration. For example, something like this would be nice to have:

$ nix profile install nixpkgs#cowsay
$ nix profile export manifest.json
# On another system:
$ nix profile import manifest.json
# Or:
$ nix shell --manifest manifest.json
# Or in home-manager:
{
  home.profiles.main = lib.importManifest ./manifest.json;
}

This is similar to what Guix is already doing: https://guix.gnu.org/manual/devel/en/html_node/Writing-Manifests.html

[bobvanderlinden]

Now that #7788 has been merged, thia issue as described is resolved. The title of the issue still applies though 😅

[alper]

warning: Use 'nix profile list' to see the current profile

Is not a particularly useful message.

[expelledboy]

I still feel the upgrade experience is terrible...

~/repos/expelledboy/erl2json on  master ⌚ 12:53:00
$ nix --version
nix (Nix) 2.13.3

~/repos/expelledboy/erl2json on  master ⌚ 12:53:02
$ nix profile list
0 github:expelledboy/erl2json#defaultPackage.aarch64-darwin github:expelledboy/erl2json/781aedb20efb4a4843666dc23ac6ed310093bbf0#defaultPackage.aarch64-darwin /nix/store/88zgy8kwv210rhg5vybzsclw0b6r3f43-erl2json-20230523041942-781aedb
1 - - /nix/store/mymr7br3r8x0fcr0l144vbhlhx4zg8rl-home-manager-path

~/repos/expelledboy/erl2json on  master ⌚ 12:53:05
$ nix profile install github:expelledboy/erl2json
error: files '/nix/store/88zgy8kwv210rhg5vybzsclw0b6r3f43-erl2json-20230523041942-781aedb/bin/erl2json' and '/nix/store/2hr90l00pz1xf38xjkn4pjr5sbgpc71n-erl2json-20231213171651-08d30ac/bin/erl2json' have the same priority 5; use 'nix-env --set-flag priority NUMBER INSTALLED_PKGNAME' or type 'nix profile install --help' if using 'nix profile' to find out how to change the priority of one of the conflicting packages (0 being the highest priority)

~/repos/expelledboy/erl2json on  master ⌚ 12:53:09
$ nix profile upgrade github:expelledboy/erl2json
warning: 'github:expelledboy/erl2json' does not match any packages
warning: Use 'nix profile list' to see the current profile.

~/repos/expelledboy/erl2json on  master ⌚ 12:53:13
$ nix profile upgrade 0
error: flake 'github:expelledboy/erl2json' does not provide attribute 'defaultPackage.aarch64-darwin'

[bobvanderlinden]

Related issue: #7960
Possible solution: #8678

[pmarreck]

It's funny because my ixnay wrapper (which I started writing and using precisely because of the original nonsense shown in the earlier part of this ticket) stopped working for upgrading profiles and I arrived here while googling why. Turns out I had written all this code to parse out the correct information given profile numbers including multiple installs of the same package (eyeroll) and you guys went and fixed the TUI UX. ;) So now I have to update my wrapper script, but I'm happy to do so!

I use NixOS exclusively, except for when I use Nix on Darwin... so unfortunately I need the imperative-install thing (I suppose I could also use home manager darwin, but not ready to make that leap yet.)