[FR] Add support for PEP 643 - metadata for sdists

[pganssle]

What's the problem this feature will solve?

Some time ago we approved PEP 643, and I intended to implement it... eventually, but it's been a long time and this is actually a very powerful and useful tool for setuptools in particular to implement.

If done correctly, we should be able to silently and in the background start providing reliable metadata for source distributions to anyone building with new versions of setuptools.

Describe the solution you'd like

On the install/wheel building side, we need to be aware that if we are consuming an sdist that already has version 2.2+ metadata, we take all non-dynamic values from the sdist, and not from setup.py. That way if setup.py and the sdist metadata disagree about a given non-dynamic value, the sdist wins (and we are in compliance with the PEP).

For generating PEP 643 metadata (the bigger and more important job, I imagine), there are several implementations (and they can be progressively and backwards-compatibly implemented):

1. The most naïve implementation: mark everything as Dynamic and bump the version to 2.2. (This is not a very useful implenentation, but a valid one).
2. Mark everything as Dynamic if it comes from setup.py, but not if it comes from setup.cfg or pyproject.toml (when [FR] support pep-621 - storing core project metadata in pyproject.toml #2671 is resolved).
3. Add a mechanism to explicitly mark a value as static or dynamic in setup.py — a wrapper like setuptools.literal_value, for when someone needs to generate something from code, but it's deterministic for any given release. We don't have to worry too much about this being inaccurate if we also ensure that when setup.py and the sdist metadata disagree, the final value from the wheel / installed version is taken from the metadata.
4. Add heuristics to parse the AST of setup.py and determine when setup() is passed arguments that we can be sure are deterministic. Literal values, things not assigned in if blocks, that sort of thing. I suspect that even a small amount of this sort of thing will allow us to unlock a lot of currently deterministic metadata that we'll otherwise have to mark as dynamic, e.g. anything with a setup.py that looks like this:

setup(
    name="foo",
    version="0.0.1",
    install_requires = ["attrs"],
)

Alternative Solutions

No response

Additional context

See the discussion on the discourse.

Code of Conduct

• I agree to follow the PSF Code of Conduct

[domdfcoding]

This will ultimately need pypi/warehouse#9660 to be merged first, otherwise the sdists won't be uploadable to PyPI.

[GalaxySnail]

FWIW, metadata 2.2 has been supported in warehouse: pypi/warehouse#13606

[abravalheri]

The outcome of the discussion in #4629 seems to indicate the following is a viable approach:

1. Document that plugins are not allowed to dynamically generate inconsistent values for name and version metadata (they may generate these, but the values need to be exactly the same across all builds from both source tree or sdist)
2. Document that plugins are not allowed to modify values provided statically as metadata in pyproject.toml
3. When metadata is provided by pyproject.toml, setuptools will mark a field as dynamic only if it is marked as dynamic in pyproject.toml, following PEP 621
4. When metadata is provided via setup.py, setuptools will add Dynamic for the metadata fields to ensure other tools do not accidentally trust them as static (just in case).

Originally posted in #4629 (comment)

[abravalheri]

#4698 is an attempt to implement PEP 643. I am trying a slightly unorthodox approach by marking the static values coming from setup.cfg/pyproject.toml instead of the dynamic ones, in an attempt to identify when plugins/customisations overwrite them. I say the approach is unorthodox because I introduce sub-classes of built-in data types. That is a bit awkward, but it makes sure the change is backwards compatible with the existing codebase in both setuptools and distutils.

If anyone is interested in having a look on the approach, please let me know your ideas.