Upgrade vendored packaging lib

[sbidoul]

Upgrade our vendored copy of packaging, remove all references to LegacyVersion and LegacySpecifier.

Closes #12063
Closes #11715
Closes #11445
Closes #7650
Closes #9613
Closes #10098

TODO

• handle resolution with candidates with invalid versions
• handle resolution with candidates with invalid requirements
• handle environments containing installed distributions with legacy versions
• handle environments containing installed distributions with invalid specifiers in their dependencies

[sbidoul]

@pypa/pip-committers this is a quick try at upgrading packaging.

The tests that need fixing may reveal a few interesting things.

In particular I wonder if we should worry about 920c4fc. When the pip-test-package-0.1.tar.gz sdist is in the find-links path, pip now fails with Invalid version test-package-0.1.

[sbidoul]

And the last failing test (test_install_special_extra[Hop_hOp-hoP-hop-hop-hop]) eludes me for now. It succeeds on 3.11 and 3.12 but fails on 3.10 and below.

[pradyunsg]

I just tried a couple of quick changes to the pkg_resources metadata backend to make it respect PEP 685 normalisation. A quick check reflects that this does behave correctly -- I'll wait for the CI check on this though, as well as a round of reviews before moving forward with it.

[pradyunsg]

Quoting @pfmoore from the discussion above...

The problem is that setuptools.DistInfoDistribution makes the extras attribute a list of extras normalised by safe_extra. But pip has canonicalised the requested extra, and it can't re-constitute the non-canonical value that's in extras (nor should it). I don't think this is easily fixable without a fairly extensive rewrite of our extra handling (or a fix to pkg_resources to store normalised extra names, which, to be fair, isn't required by PEP 685).

Ugh, this is a nuance that I missed digging into this head-first. Indeed, dependency_map[safe_extra(ext)] is how pkg_resources handles names. :/

[pfmoore]

Nice! Looks like you may have fixed it @pradyunsg 🙂 I hadn't thought to go the whole way and manage getting the extras from the metadata directly (effectively copying the importlib.metadata approach for pkg_resources). That's a much more localised change than I'd feared we might need.

[pradyunsg]

Ohkie, this last commit is an attempt at simplification -- we might want to leave it out if it isn't all-green, since it's unlikely I'd get a chance to look at this again before Monday morning.

[pradyunsg]

I've taken the liberty to mark this as ready for review.

[pradyunsg]

As I do that though... I'm realising that one thing that we ought to validate is whether there are suboptiomal behaviours/failure modes, when given a legacy version or specifier; coming via the index page, a built wheel, a project's metadata and various other data sources that pip interacts with.

[sbidoul]

This is not ready indeed. There is work to do to ignore invalid versions in indexes (e.g. pip install pytz fails because there are old invalid versions on PyPI), and ignore distributions with invalid version specifiers in their dependencies. And handle invalid installed versions. And probably more...

[notatallshaw]

Other fails include pip install nltk, pip install apache-airflow[all], and pip install pandas (I raised similiar issues with rip recently which I beleive also vendors the latest version of Packaging: prefix-dev/rip#27, prefix-dev/rip#38, and prefix-dev/rip#25)

FYI in PR #12327 I am currently filtering out invalid file name and version name packages as Pip collects them in that particular scenario, not sure if that's the going to be the prefered way to do it.

[sbidoul]

Yeah, I now remember why I said I would not have time to do this before the release. I somehow had managed to forget about #12300 (comment) when I created this PR.

This seems to be way too far reaching to do so late in the release cycle. So I think I'll postpone it.

[pfmoore]

There is work to do to ignore invalid versions in indexes (e.g. pip install pytz fails because there are old invalid versions on PyPI), and ignore distributions with invalid version specifiers in their dependencies. And handle invalid installed versions. And probably more...

Are there issues tracking these (that we could put into the 24.0 milestone)? If we don't have some means of tracking what's needed to get this landed, I fear we're going to be in the same situation next release - in spite of no-one liking it, vendoring upgrades tend to get ignored until late in the release cycle...

[tacaswell]

This breaks pip on the CPython maintenance branches as the version reported by Python ends with a + and nothing else

ERROR: Exception:
Traceback (most recent call last):
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_internal/commands/install.py", line 377, in run
    requirement_set = resolver.resolve(
                      ^^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
    result = self._result = resolver.resolve(
                            ^^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 427, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 239, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 229, in _get_updated_criteria
    for requirement in self._p.get_dependencies(candidate=candidate):
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 247, in get_dependencies
    return [r for r in candidate.iter_dependencies(with_requires) if r is not None]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 243, in iter_dependencies
    for r in requires:
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_dists.py", line 222, in iter_dependencies
    elif not extras and req.marker.evaluate({"extra": ""}):
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/packaging/markers.py", line 252, in evaluate
    return _evaluate_markers(self._markers, current_environment)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/packaging/markers.py", line 158, in _evaluate_markers
    groups[-1].append(_eval_op(lhs_value, op, rhs_value))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/packaging/markers.py", line 116, in _eval_op
    return spec.contains(lhs, prereleases=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/packaging/specifiers.py", line 568, in contains
    normalized_item = _coerce_version(item)
                      ^^^^^^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/packaging/specifiers.py", line 36, in _coerce_version
    version = Version(version)
              ^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/packaging/version.py", line 200, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
pip._vendor.packaging.version.InvalidVersion: Invalid version: '3.12.3+'

Bisected to 47a8480 specifically.