meta.mainProgram missing

[davidak]

Describe the bug

For many packages, nix run does not work, because meta.mainProgram is missing.

Example error:

[davidak@gaming:~]$ nix run nixpkgs#calls
error: unable to execute '/nix/store/b1ylix7xwgknn5pypja89nh6i2s80icz-calls-43.2/bin/calls': No such file or directory

I don't want to create issues for every case and would prefer a systematic solution.

Please add any package name with this issue here or as comment!

Affected packages:

• calls
• linphone

It would also be nice to add an automatic test if meta.mainProgram exist. Maybe it should be added to nixpkgs-review?

[emilylange]

I hacked something quick together in an attempt to get a feeling how many packages are affected :)

let
  pkgs = import <nixpkgs> { };
  lib = pkgs.lib;

  dbFile = builtins.storePath <nixpkgs/programs.sqlite>;
  dbJson = pkgs.runCommand "json" { } ''
    ${lib.getExe pkgs.sqlite} ${dbFile} '.mode json' '
      SELECT package, system, json_group_array(name) AS names
      FROM Programs
      WHERE system = "x86_64-linux"
      GROUP BY package, system;
    ' > $out
  '';

  programs = lib.importJSON dbJson;
  mainProgram = x: x.meta.mainProgram or (lib.getName x);
  exe = e: mainProgram (lib.getAttrFromPath (lib.splitString "." e) pkgs);
in

pkgs.writeText "missing"
  (
    builtins.toJSON
      (lib.filter
        (
          row:
          (lib.filter
            (bin: bin == exe row.package)
            (builtins.fromJSON row.names)
          ) == [ ]
        )
        programs)
  )

❯ nix-build -I nixpkgs=channel:nixos-unstable file.nix
/nix/store/548jlinnphj8xy9mly5zpsncvb5rin1i-missing
❯ jq length result
5529

I expected it to be much worse 😅

Still, too many to fix manually imho :o

[davidak]

@IndeedNotJames i don't understand the code 😬

does it check if the file <package>/bin/<meta.mainProgram or pname or name> exist? or only if meta.mainProgram is set?
it does not need to be set when it matches the package name

nixpkgs/lib/meta.nix

Line 136 in 1c0eebb

meta.mainProgram or pname or name

do you see any chance to automate it? we would at least need to know the actual binary name. then adding it to meta might be possible

[emilylange]

Ah, sorry, will try to outline a few things:

My script relies on the programs.sqlite file, which is provided by channels, which is why -I nixpkgs=channel:nixos-unstable is needed.

It's the same database that helper scripts like command-not-found use and https://github.com/NixOS/nixos-search recently added "Programs provided" feature.
This database stores every symlink and executable in /bin/* of each evaluated package.

My

mainProgram = x: x.meta.mainProgram or (lib.getName x);

function mimics the lib.getExe you linked to, just without the /nix/store/*/bin/ prefix.

I kinda just expected nix run to use lib.getExe under the hood, but turns out it doesn't¹ 👀

Assuming I didn't misunderstand the c++ and nix code, it still follows the same meta.mainProgram or pname or name logic as lib.getExe²³

My script does not differentiate if meta.mainProgram or pname or name is set.
It just loops through each package and checks if whatever my trimmed down version of lib.getExe (mainProgram) exists.
It could very well be that some packages have meta.mainProgram set, that are wrong or outdated.

---

Take for example ansible:

{
  "names": "[\"ansible-vault\",\"ansible-test\",\"ansible-pull\",\"ansible-playbook\",\"ansible-inventory\",\"ansible-galaxy\",\"ansible-doc\",\"ansible-console\",\"ansible-connection\",\"ansible-config\",\"ansible\"]",
  "package": "ansible",
  "system": "x86_64-linux"
},
{
  "names": "[\"ansible-vault\",\"ansible-test\",\"ansible-pull\",\"ansible-playbook\",\"ansible-inventory\",\"ansible-galaxy\",\"ansible-doc\",\"ansible-console\",\"ansible-connection\",\"ansible-config\",\"ansible\"]",
  "package": "ansible_2_12",
  "system": "x86_64-linux"
},
{
  "names": "[\"ansible-vault\",\"ansible-test\",\"ansible-pull\",\"ansible-playbook\",\"ansible-inventory\",\"ansible-galaxy\",\"ansible-doc\",\"ansible-console\",\"ansible-connection\",\"ansible-config\",\"ansible\"]",
  "package": "ansible_2_13",
  "system": "x86_64-linux"
}

My script lists 3 ansible variants.
ansible is an alias to whatever the current ansible_* version is, which in turn points to python3Packages.ansible-core⁴.

ansible (not ansible_2_12 or ansible_2_13) would suggest a match of the ansible binary in names[] (because attribute name [not drv.name] == binary name).

mainProgram pkgs.ansible, however, yields ansible-core, because the aliased pname is called ansible-core.

lib.getExe pkgs.ansible yields the same result, e.g. /nix/store/7fnn5sh81kjc2va4xhl175cv929hqyh1-python3.10-ansible-core-2.14.2/bin/ansible-core.

There are also a lot of lispPackages.* my script returns.
E.g. lispPackages._1am -> _1am-lisp-launcher.sh.
I am not familiar with lisp, but it seems like this comes from

nixpkgs/pkgs/development/lisp-modules/define-package.nix

Line 101 in 1c0eebb

"$out/bin/${args.baseName}-lisp-launcher.sh"

Which seems like a very easy fix to add mainProgram to

nixpkgs/pkgs/development/lisp-modules/define-package.nix

Lines 114 to 116 in 1c0eebb

	meta = {
	inherit description version;
	} // meta;

❯ jq 'map(select(.package | startswith("lispPackages."))) | length' result
388

---

I am afraid this response made everything more confusing 😅

But I do strongly agree that some sort of check should be added to borg, nixpkgs-review or whatever. Something that fits in the current workflow, raises awareness.

do you see any chance to automate it? we would at least need to know the actual binary name. then adding it to meta might be possible

Some automated way would be awesome, yeah. No idea how to catch all those weird edge cases, though (yet?)

But there are also A LOT of packages that just have a single binary in their /bin/*, which would be trivial to fix:

❯ jq 'map(select((.names | fromjson  | length) == 1)) | length' result
2473

Footnotes

1. https://github.com/NixOS/nix/blob/523913d091a0c090edb81d68240695b8f50f5b98/src/nix/app.cc#L89-L111 ↩

2. https://github.com/NixOS/nixpkgs/blob/1c0eebb1d64f4e07d4bea458b3824b959beb12d4/lib/meta.nix#L146-L148 ↩

3. https://github.com/NixOS/nixpkgs/blob/1c0eebb1d64f4e07d4bea458b3824b959beb12d4/lib/strings.nix#L638-L643 ↩

4. https://github.com/NixOS/nixpkgs/blob/1c0eebb1d64f4e07d4bea458b3824b959beb12d4/pkgs/top-level/all-packages.nix#L17172-L17188 ↩

[emilylange]

cc @7c6f434c, @Uthar, @alyssais

As said above, I am not familiar with lisp, but would setting meta.mainProgram to *-lisp-launcher.sh make sense?
Would anyone ever run lispPackages.* with nix run or maybe benefit from it differently?

And what about lispPackages_new.*?

❯ jq 'map(select(.package | startswith("lispPackages_"))) | length' result
38

[emilylange]

Related #246386

[Artturin]

With #246386 we have eval time warnings instead of build time failures

[roberth]

nix run does not work, because meta.mainProgram is missing.

For nix run this should be a thing of the past as newer versions of nix assume ${pname} is the main program.
See post for design considerations.
That's part of a discussion about lib.getExe behavior, for which such logic was soft reverted in

• lib.getExe: Do not make assumptions about the main program #246386

This design choice makes the addition of meta.mainProgram values relevant again, but for a subtly different reason.

[luochen1990]

I'm also having this issue:

Package "python3.10-jc-1.23.4" does not have the meta.mainProgram attribute

[Fryuni]

Overlay to remove the warning while the generating script is not changed:

_: pkgs:
{
  xorg = pkgs.xorg // {
    xrandr = pkgs.xorg.xrandr.overrideAttrs (_: prev: {
      meta = prev.meta // {
        mainProgram = "xrandr";
      };
    });
  };
}

[Artturin]

Overlay to remove the warning while the generating script is not changed:

_: pkgs:
{
  xorg = pkgs.xorg // {
    xrandr = pkgs.xorg.xrandr.overrideAttrs (_: prev: {
      meta = prev.meta // {
        mainProgram = "xrandr";
      };
    });
  };
}

Can be fixed here

nixpkgs/pkgs/servers/x11/xorg/overrides.nix

Line 978 in 06b25e9

xrandr = super.xrandr.overrideAttrs (attrs: {

[Fryuni]

Uuu, I didn't see that. Nice.

I think having this on the overrides would still be a temporary workaround. I think the script that generates the this file (which I also haven't found) should include the name of everything that is an executable.

But if that is not feasible or the cost/benefit doesn't justify than having those are overrides is fine.

I'll open a PR for xrandr and link it here so at least the warning goes away.

And I notice that I sent the comment on the wrong tab 😅 . At least is on a related issue.

[KiaraGrouwstra]

it does not need to be set when it matches the package name

that no longer seems to be what the warning implies now:

warning: getExe: Package "consul-alerts-0.6.0" does not have the meta.mainProgram attribute. We'll assume that the main program has the same name for now, but this behavior is deprecated, because it leads to surprising errors when the assumption does not hold. If the package has a main program, please set meta.mainProgram in its definition to make this warning go away. Otherwise, if the package does not have a main program, or if you don't control its definition, use getExe' to specify the name to the program, such as lib.getExe' foo "bar".

[Atemu]

Fixed by #297084