feat(pip): allow direct reuse of upstream derivations

[yajo]

Sometimes I'm just interested in the targets part of pip locking system. However, if a derivation can be found directly in nixpkgs, I prefer to use that one instead of building it downstream with dream2nix.

This is because I can then benefit from upstream maintenance, patching and caching done by nixpkgs directly.

Thus, I'm here adding a preferredDrvs option to pip module. You can add the derivations that you want to search by name. If found and requested by the resulting python closure, the derivation will be used directly instead of being auto-generated.

Usually you can just pass config.deps.python.pkgs here and avoid rebuilding a lot of stuff automatically. It should help on avoiding duplicate dependencies too.

I have not added any tests because I'd like to get some feedback over this prototype before testing.

@moduon MT-1075

[phaer]

Something similar has been discussed before - see nixpkgs-overrides. I think this approach has at least one pitfall which doesn't seem to get handled here (?): Transitive dependencies of the drv you are including from nixpkgs are included as well - no matter whether they match or not.

[yajo]

OK, let me see if I can just fix the same scenario with nixpkgs-overrides.

[yajo]

After trying for awhile, this is definitely different.

To start, I opened #948 which is required for unstable these days. However, even after merging that, nixpkgs-overrides tries to respect some data from the lock file to actually do a merge between what's done in nixpkgs and what's in the lock file. Attributes that are kept by default and use the values from the lock file:

dream2nix/modules/dream2nix/nixpkgs-overrides/default.nix

Lines 67 to 76 in 6b8b4a8

	nixpkgs-overrides.exclude = [
	"all"
	"args"
	"builder"
	"name"
	"pname"
	"version"
	"src"
	"outputs"
	];

The alternative I'm proposing here, instead, gets the exact same derivation that's in nixpkgs. Effectively speaking, it would ignore anything in the lock file as long as the derivation is already built and supported in nixpkgs. It would be like overlaying only the missing derivations on top of nixpkgs. It renders some of the work done by the fetch-and-lock process useless, but also saves some headaches if you have a ton of dependencies that require manual patches.

However, after testing some more, I just found out that I could do this with a very simple module:

# nixpkgs-reuse.nix
{dream2nix,  lib,  config,  ...}: {
  imports = [dream2nix.modules.dream2nix.nixpkgs-overrides];
  package-func.func = lib.mkForce (lib.const config.nixpkgs-overrides.from);
}

I could apply it only to the dependencies I want from nixpkgs:

{dream2nix, lib, config, ...}: {
  imports = [dream2nix.modules.dream2nix.pip];
  pip.drvs = {
    requests = ./nixpkgs-reuse.nix;
    setuptools = ./nixpkgs-reuse.nix;
  };
}

Or I could even auto-overlay anything from nixpkgs:

{dream2nix, lib, config, ...}: {
  imports = [dream2nix.modules.dream2nix.pip];
  pip.drvs = lib.foldlAttrs (acc: name: {type, ...}:
    acc
    // lib.optionalAttrs (type == "url" && config.deps.python.pkgs ? ${name}) {
      ${name} = ./nixpkgs-reuse.nix;
    }) {} config.lock.content.fetchPipMetadata.sources;
}

My context is a big pseudo-monorepo that consists on a lot of python packages. I'm trying to build local dependencies and any non-local ones that are not in nixpkgs, while trusting in that upstream distribution otherwise.

I'll close this PR, as this one seems to be a good solution for me. Thanks!