Allow packages to point outside their directory

[philiptaron]

For derivations that wrap things in nixpkgs itself -- for instance, the NixOS manual and the Nixpkgs manual -- the rule that paths are found only in the by-name directory itself is too restrictive. It's a good default, but there should be an opt-out mechanism.

I ran into this implementing NixOS/nixpkgs#311459.

[infinisil]

A big motivation for this is avoiding action at a distance. Having this restriction makes it very clear that nothing outside the package directory can influence its result.

And specifically in the case of the manuals, it hints at a deeper problem: The fact that the top-level Nixpkgs value (import <nixpkgs> {}) contains the packages directly, which gives no fitting space for non-package attributes. Nixpkgs doesn't follow the rule that the top-level value should be a record, and that's a problem.

The only solution for that is to create a separate top-level value that doesn't have this issue. Perhaps something like:

import <nixpkgs> {
  apiVersion = 1;
}

[teto]

I also get the error with NixOS/nixpkgs#337033, see output at:
https://github.com/NixOS/nixpkgs/actions/runs/10539567151/job/29203277988?pr=337033

I dont know if action at distance is a valid concern here, the dependency is explicit.
cp ${../../../../maintainers/scripts/pluginupdate.py} $out/lib/pluginupdate.py.

What's your advice ?

Note: I wish the nixpkgs CI would give more pointers to help understand the message like sharing the link to this repo. Each package directory must not refer to files outside itself using symlinks or Nix path expressions. is also a bit frustrating to read without a rationale behind it (that you provide here, and to which I can agree).

[infinisil]

Oh I actually talked about this at NixCon: https://media.ccc.de/v/nixcon-2023-35713-not-all-packages-anymore-nix#t=742

Summarising and expanding:

• The package directories should be independent, so we can move them around later (e.g. when we change the hashing schema, move to independent repositories, or move them to another directory)
• Accessing arbitrary files is not great because it discourages us from restructuring Nixpkgs, since moving any file could break a package in an arbitrary other place
• Instead of accessing such files, I recommend adding an extra callPackage argument to pass in the file that you need explicitly, therefore also allowing it to be overridden with .override.

For your specific case @teto, it sounds like pluginupdate.py could benefit from being packaged as pkgs.pluginupdate or perhaps pkgs.internal.pluginupdate to indicate that this is not a package meant to be installed by users. Then you can access it in your package with the standard callPackage arguments. Alternatively, do callPackage .../package.nix { pluginupdate = .../pluginupdate.py; } in all-packages.nix.

[infinisil]

Opened #97 to improve the error :)

[teto]

thanks for the quick feedback and the pointers. I've used your last trick: Alternatively, do callPackage .../package.nix { pluginupdate = .../pluginupdate.py; } in all-packages.nix..

I had tried beforehand to add a symlink towards pluginupdate.py in the same folder but this tool caught the hack so good job ^^''