Replace `command-not-found' with nix-index

[matthewbauer]

@bennofs has created a useful project called nix-index which provides an index of files built by Hydra

https://github.com/bennofs/nix-index

In many ways it is similar to command-not-found.pl. Some advantages of nix-index, though:

• works outside of NixOS
• has indexes for ALL files, not just those in /bin/
• written in Rust instead of Perl

I'm wondering whether people think this would be worthwhile to do? Instead of shipping channels with programs.sqlite we could include the generated nix-index. It would probably be slightly larger but also much more versatile.

[coretemp]

Deprecating command-not-found.pl after it has been established that nix-index is better for the existing use case is 👍 for me.

First add a default off NixOS module which uses nix-index instead of command-not-found.pl, collect some data about usage (for example, your own), and then use that to build the case for actually making that optional module the default.

[Mic92]

Quoting myself from #88590

Our command-not-found depends on channels, which is soon replaced by flakes and no longer contain the sqlite database. An alternative could be to ship nix-index which has a similar functionality. However its index is comparable large (27mb). As an alternative we could split of all filenames in ./bin into its own database and ship it independently. As an experiment I exported this sqlite database to csv and than compressed it with bzip2. The result is 273.5K large. Users than can unlock the whole feature set of nix-index by generating their own index.

@bennofs said, he would add support for reading databases from multiple locations which would allow to have a smaller database that we install by default and a bigger one that could be installed on demand.

[stale]

I marked this as stale due to inactivity. → More info

[wmertens]

I still think this would be good, even with a 27MB database (since it can easily be disabled for minimal images). With a 270KB database, it's a no-brainer.

[stale]

I marked this as stale due to inactivity. → More info

[Mic92]

still relevant.

[LunNova]

For those who found this from search and want to use nix-index and already use home-manager, programs.nix-index.enable = true; is all you need.

https://nix-community.github.io/home-manager/options.html#opt-programs.nix-index.enable

If you would also like to pin your system nixpkgs to the one in your flake.nix, it seems to be as easy as nix.nixPath = [ "nixpkgs=${nixpkgs}" ];. Not quite the same as setting the channel but I don't see a way to do that.

[virchau13]

nix-index is much slower than command-not-found:

<hexagon:~>% time command-not-found 'yt-dlp'
The program 'yt-dlp' is not in your PATH. It is provided by several packages.
You can make it available in an ephemeral shell by typing one of the following:
  nix-shell -p python38Packages.yt-dlp
  nix-shell -p python38Packages.yt-dlp-light
  nix-shell -p python39Packages.yt-dlp
  nix-shell -p python39Packages.yt-dlp-light
command-not-found 'yt-dlp'  0.03s user 0.01s system 97% cpu 0.044 total
<hexagon:~>% time nix-locate --whole-name 'bin/yt-dlp'
yt-dlp-light.out                                    651 x /nix/store/ny548a35xyn61n9sinmwnck74bwh7kng-python3.9-yt-dlp-2021.10.22/bin/yt-dlp
yt-dlp.out                                          714 x /nix/store/r1anad4h6xhffzrpzd2fdmph6qhp687b-python3.9-yt-dlp-2021.10.22/bin/yt-dlp
python38Packages.yt-dlp-light.out                   652 x /nix/store/7grwipnamfhc6k3cdlc7lp134icf4qzv-python3.8-yt-dlp-2021.10.22/bin/yt-dlp
python38Packages.yt-dlp.out                         715 x /nix/store/l418f9jv12c427cwna6pr1xjx7dnhiyn-python3.8-yt-dlp-2021.10.22/bin/yt-dlp
nix-locate --whole-name 'bin/yt-dlp'  1.15s user 0.05s system 99% cpu 1.201 total

This makes sense, since nix-index has to store a lot more information and support partial searches, but it makes nix-index difficult to use as a "command not found" handler. Since command-not-found can get away with a much less sophisticated implementation and be much faster, I believe command-not-found and nix-index should be separate. (command-not-found will probably have to be updated to work with flakes, however.)

[Mic92]

nix-index is much slower than command-not-found:

<hexagon:~>% time command-not-found 'yt-dlp'
The program 'yt-dlp' is not in your PATH. It is provided by several packages.
You can make it available in an ephemeral shell by typing one of the following:
  nix-shell -p python38Packages.yt-dlp
  nix-shell -p python38Packages.yt-dlp-light
  nix-shell -p python39Packages.yt-dlp
  nix-shell -p python39Packages.yt-dlp-light
command-not-found 'yt-dlp'  0.03s user 0.01s system 97% cpu 0.044 total
<hexagon:~>% time nix-locate --whole-name 'bin/yt-dlp'
yt-dlp-light.out                                    651 x /nix/store/ny548a35xyn61n9sinmwnck74bwh7kng-python3.9-yt-dlp-2021.10.22/bin/yt-dlp
yt-dlp.out                                          714 x /nix/store/r1anad4h6xhffzrpzd2fdmph6qhp687b-python3.9-yt-dlp-2021.10.22/bin/yt-dlp
python38Packages.yt-dlp-light.out                   652 x /nix/store/7grwipnamfhc6k3cdlc7lp134icf4qzv-python3.8-yt-dlp-2021.10.22/bin/yt-dlp
python38Packages.yt-dlp.out                         715 x /nix/store/l418f9jv12c427cwna6pr1xjx7dnhiyn-python3.8-yt-dlp-2021.10.22/bin/yt-dlp
nix-locate --whole-name 'bin/yt-dlp'  1.15s user 0.05s system 99% cpu 1.201 total

This makes sense, since nix-index has to store a lot more information and support partial searches, but it makes nix-index difficult to use as a "command not found" handler. Since command-not-found can get away with a much less sophisticated implementation and be much faster, I believe command-not-found and nix-index should be separate. (command-not-found will probably have to be updated to work with flakes, however.)

Is this not just because the current index is bigger? If nix-index would use a smaller database that just contains $out/bin the speed is probably similar.

[virchau13]

Is this not just because the current index is bigger?

Hmm... that appears to be true. nix-locate -r '.*' | wc -l outputs 15748730 ≈ 1.6 x 10^7 entries, whereas nix-locate -w -r '^/bin/.*$' | wc -l outputs 131175 ≈ 1.3 x 10^5 entries, which indicates a roughly 120x speedup.

This change makes sense, then, especially if the /bin database is only 270K.

[wmertens]

Since we're on the subject, how does nix-locate get the db? C-n-f gets it via the channel, but that is not available in flakes. It would be nice to have a mapping from Hydra build commitishes to nix-locate-bin-only output hashes

[virchau13]

I decided to implement prefix filtering in nix-index to test.

Here is the nix-index database with only files matching /bin/* added:

@nix-index <hexagon:~/prog/repos/nix-index>% du -sh data/files
2.8M	data/files

The nix-index database is only 2.8MiB, compared to the 5.1MiB needed for command-not-found's database.

@nix-index <hexagon:~/prog/repos/nix-index>% time nix-locate --at-root -w '/bin/yt-dlp' --db data/
yt-dlp.out                                          713 x /nix/store/57sxxkq50rpkrww2s8j7y316avg90f1h-python3.9-yt-dlp-2022.2.4/bin/yt-dlp
yt-dlp-light.out                                    648 x /nix/store/c0zg1q1dv17shdmchkzhvpj1mq6dfwl7-python3.9-yt-dlp-2022.2.4/bin/yt-dlp
python310Packages.yt-dlp.out                        717 x /nix/store/9snrnr6wxpyzylzmqbq6dm9lg6gpyg8d-python3.10-yt-dlp-2022.2.4/bin/yt-dlp
python310Packages.yt-dlp-light.out                  652 x /nix/store/appg4gd91hq32425p4ran8ajs27dxa0p-python3.10-yt-dlp-2022.2.4/bin/yt-dlp
nix-locate --at-root -w '/bin/yt-dlp' --db data/  0.03s user 0.01s system 97% cpu 0.039 total

It works at about the same speed.

Overall, I would say using nix-index is definitely a good idea.

Here is what needs to happen on the nix-index side of things:

• Get nix-index to use flakes to generate the database. This shouldn't be too difficult, as all nix-index needs is a path to nixpkgs. It's already possible to make nix-index work by using nix.nixPath = [ "nixpkgs=${inputs.nixpkgs}" ]; in a flake system configuration.
• Allow changing of the nix-index cache server (right now it is hardcoded to cache.nixos.org). This is also an easy change.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/why-isnt-there-an-official-built-in-way-to-find-what-package-provides-a-specific-executable/22937/3