python3.pkgs.buildPythonPackage: Separate runtime & build time dependencies

[adisbladis]

Description of changes

See #272178 for more context.

Dependency separation

Adds a separation between Python runtime dependencies, Python build-time dependencies & non-Python dependencies arguments to buildPythonPackage.
It's important to have these distinctions so we can solve our long standing issues re dependency propagation in Python.
With this change in place it's possible to implement a solution for #170577.

It's similar to #102613 (a subset, really) but with the notable changes that:

• Uses PEP-621 standard names/idioms
  WIP: Python: wrap and patch using requiredPythonModules instead of propagatedBuildInputs #102613 introduces the argument requiredPythonModules, but with Python metadata being standardised I'd like to be more closely aligned with that.
• Only implements the API that enables changing the behaviour
  It's probably going to take a while to refactor all of nixpkgs to use the appropriate arguments.
  This change is purposely minimal in the hopes that we can move forward with it more quickly.

For now dependencies are always mapped onto propgatedBuildInputs, so there is no change in behaviour in the immediate term.
2 packages are migrated as a PoC, one is using format = "pyproject" and the other is setuptools.

Interface changes

This also unifies regular dependencies with optional dependencies:

{
  passthru.optional-dependencies.socks = [ pysocks ];
  nativeBuildInputs = [ poetry-core ];
  propagatedBuildInputs = [ requests ];
}

becomes

{
  optional-dependencies.socks = [ pysocks ];
  build-system = [ poetry-core ];
  dependencies = [ requests ];
}

which looks much more like what a standard pyproject.toml file contains.
I think the interface change alone is a pretty good improvement.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

Priorities

Add a 👍 reaction to pull requests you find important.

[jonringer]

I wonder if it would be better to just expose a new mkDerivation helper (e.g. buildPyprojectPackage). I think within this context, it's simpler to expose the PEP-621terminology without having to compound it with the legacy behavior of buildPythonPackage.

[adisbladis]

I wonder if it would be better to just expose a new mkDerivation helper (e.g. buildPyprojectPackage). I think within this context, it's simpler to expose the PEP-621terminology without having to compound it with the legacy behavior of buildPythonPackage.

As discussed on Matrix I don't like adding another function because:

• The naming is already confusing between buildPythonPackage/buildPythonApplication

  This would be even more murky with a third name thrown in the mix.

• This PR doesn't semantically change much, it just allows you to speak about Python dependencies specifically as opposed to explaining the Nix semantics explicitly like propagatedBuildInputs.

  buildPythonPackage has no real notion of Python dependencies which is a bit silly when you think about it.

• Python builds are converging on pyproject

  PEP-517 style builds are becoming the default in the Python ecosystem. It would be unfortunate if our default Python builder wasn't aligned with the default Python package formats.

• Optional dependencies

  Optional dependencies are currently "supported" by buildPythonPackage by the ad-hoc use of passthru.
  This PR makes optional-dependencies (setuptools calls this extras) an explicit concept known to buildPythonPackage.
  These are not new concepts, PEP-621 just gave them better more clear naming.

[SuperSandro2000]

• The naming is already confusing between buildPythonPackage/buildPythonApplication

💯
I have done a lot with nixpkgs python and it is still hard for me to remember which I am supposed to use.

---

Also if we can get away with not fragmenting things and just shove this under the current system without breaking everything then I would be super happy.

[FRidh]

The main topic here is the separation. We need that, without a doubt.

Then there is the naming. This PR introduces non-standard names for Nixpkgs, but names that are the standard in Python packaging. While I've long held the opinion that we should have names that are similar to what is used elsewhere in Nixpkgs, this builder is used by other higher-level tools such as dream2nix and pdm2nix, which would preferably work with an interface more closely aligned to Python packaging, and for direct users of this function it is I think also more convenient.

I'd say let's go ahead with this.

[natsukium]

I have no particular opinion on the naming and have left some comments about the document.

Then, not just for this PR, but I would like to see a section like the changelog.
Since not everyone follows these discussions and changes, it would be easier to catch up if we had something to list all the changes.

[adisbladis]

Rebased, refactored & fixed comments.

[uninsane]

on the Nix side, i feel like we've spent a good deal of effort pushing language-specific logic away from mkDerivation wrappers and toward things like setup hooks. in other threads, i'm now being asked to change my packages from:

nativeBuildInputs = [ poetry-core ];

to

build-system = [ poetry-core ];

isn't this a regression? as far as i can tell, build-system is some bespoke extra concept every future nixpkgs packager will have to learn once they encounter Python. either we should rework the packaging for every other language and use build-system = [ cmake ];, build-system = [ meson ];, to regain the uniformity we had before this change, or the benefit from this is really so great as to outweigh the increased load it puts on first-time packagers.

[mweinelt]

isn't this a regression?

No, a conscious decision to adopt the PEP518/621 lingo in the Python-specific builders.

[uninsane]

No, a conscious decision to adopt the PEP518/621 lingo in the Python-specific builders.

it's not clear to me reading just this PR thread if there is wide agreement in adopting python-specific lingo at this level within nixpkgs. at least, in the handful of people who weighed in here, the issue of naming/jargon was mentioned more than once. honestly, i don't mean to bikeshed, and if this was discussed more thoroughly in a different thread then link me there and i'll be quiet. yet i think it has to be acknowledged that this has real repercussions (some bad, some good) for first-time packagers. new contributors have a hard enough time writing their first package. it's made worse if none of the knowledge learned in that first package carries through to their second package.

[natsukium]

or the benefit from this is really so great as to outweigh the increased load it puts on first-time packagers.

Yes, this change would significantly benefit nixpkgs.
It is described in several issues and PRs linked to this PR.

Adding Python modules to the nativeBuildInputs, buildInputs, propagatedBuildInputs, and nativeCheckInputs that are commonly used in current nixpkgs has caused problems for years.
To solve this, we decided to introduce new parameters to handle inputs for Python.

We did not change nativeBuildInputs -> build-system and propagatedBuildInputs -> dependencies just to follow standard Python naming.
We introduced the new concepts and referenced PEP518/621 for the names.
Honestly, I still don't care about parameter names, but I think it's a good decision to follow them now that Python packaging is converging to PEP517/518/621, unlike a few years ago.

You will also encounter these concepts when using hook with mkDerivation.

I've packaged and maintained hundreds of packages, but I still can't work in an unfamiliar category without reading the manual (or the source code) though.

[dotlambda]

That's a duplicate.

Suggested change

	Aside from propagating dependencies,
	`buildPythonPackage` also injects code into and wraps executables with the
	paths included in this list. Items listed in `extras_requires` go here.

[uninsane]

fix is out for review here: #322267