April 2021 Release

[pradyunsg]

Hiya @pypa/pip-committers!

Do we want to do a pip 21.1 this month? If yes, who wants to be the RM? :)

[pfmoore]

We should, but I'm not sure if I'll have the bandwidth to be RM.

[sbidoul]

If no-one with prior experience has the bandwidth to do it I can take it.
I would do the release around the 18th or 25th of April, to be confirmed.

I've had a quick look at the procedure, which seems very clear (kudos). I have a couple of questions, though:

• I see most PR to CPython (ensurepip) that update the bundled pip also update setuptools : is that something we should do this time too ?
• At what moment should we update the vendored dependencies ? I'd rather do it not too close to the release date.
• Besides discuss.python.org, what are the places where the release needs to be announced ?

[pradyunsg]

@sbidoul Awesome! Feel free to "assign" this issue to yourself. :)

If no-one with prior experience has the bandwidth to do it I can take it.

I can do it, but I'd prefer to start spreading the knowledge of the release process among more of us.

At what moment should we update the vendored dependencies ? I'd rather do it not too close to the release date.

Do it whenever you want honestly. Closer is better IMO, but as your intuition tells you, too close is bad. :)

I see most PR to CPython (ensurepip) that update the bundled pip also update setuptools : is that something we should do this time too ?

Yea, it's usually a good idea to. It's our way of making sure that CPython's vendored wheel does not lag too far behind PyPI.

Besides discuss.python.org, what are the places where the release needs to be announced ?

This is subjective. I've usually just dropped a discuss.python.org post for most releases.

OTOH, the releases during the resolver rollout were significantly more publicised.

[pfmoore]

I see most PR to CPython (ensurepip) that update the bundled pip also update setuptools : is that something we should do this time too ?

Yes we should. With the 21.0 release, there was a problem backporting to Python 3.8 and 3.9, so 21.0 never got into those releases. The root issue is #9540, which is still open. To be honest, I don't know what to do about this, there's a mess of dependencies here with setuptools and packaging, and I don't know what (if any) movement there has been.

It's bad that we can't get the latest version of pip into Python 3.9, but we're not in control of any of the parts causing the issue. I suggested a "quick fix" here but people weren't enthusiastic, preferring to go for the more correct solution.

I'd consider it a RM call whether we want to push to get 21.1 into CPython. Personally, I'd apply the quick fix if the no-one can confirm that the "proper" fix is going to be in place in time. But I guess it's your call. I'll try to prepare a PR today with a test that pip runs with warnings as errors, and with the "quick fix" (assuming that the test is still failing!) so at least you have something to make a decision on 🙂

[pradyunsg]

We updated/can update the setuptools version, which should fix the breakage? Unless I'm missing something here, this should be fine.

But yea, that's a call for the RM to make. Lemme know if you want someone to take a closer look. I'll be happy to.

[sbidoul]

I admit I'd rather not poke this specific hornet nest myself :) So if you want to have a closer look I'll be happy if you do @pradyunsg

[pfmoore]

We updated/can update the setuptools version, which should fix the breakage?

Possibly.

I just tried to write a test for this (test_warning.py):

def test_pip_works_with_warnings_as_errors(script):
    script.environ['PYTHONWARNINGS'] = 'error'
    script.pip("--version", expect_stderr=False)

I think it should work, but right now I'm getting random test suite failures locally and I don't know why. I've just done the branch rename thing, maybe something happened there, or maybe there are test failures on main at the moment (I saw some PRs about fixing CI against main, but I think they were lint). Anyway, I don't have time to get my dev environment fixed up at the moment, so if someone else wants to add that test, feel free.

I do think that we should have the test, if only to ensure that the ensurepip PR doesn't break (because that's nearly always a last minute or after-the-fact part of the release in my experience, at which point finding out that the CPython tests fail is, shall we say, "unwelcome" 🙂)

If the test runs fine now, then I guess we might have already fixed the issue, but a quick manual test suggests not:

❯ pip --version                 
pip 21.1.dev0 from d:\work\projects\pip\src\pip (python 3.9)
PS 14:02 (.venv) [Git: main] 00:02.474 D:\Work\Projects\pip
❯ python -We -m pip --version
Traceback (most recent call last):
  File "d:\work\projects\pip\src\pip\_vendor\packaging\version.py", line 57, in parse
    return Version(version)
  File "d:\work\projects\pip\src\pip\_vendor\packaging\version.py", line 298, in __init__
    raise InvalidVersion("Invalid version: '{0}'".format(version))
pip._vendor.packaging.version.InvalidVersion: Invalid version: 'setuptools'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Gustav\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\Gustav\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "d:\work\projects\pip\src\pip\__main__.py", line 21, in <module>
    from pip._internal.cli.main import main as _main
  File "d:\work\projects\pip\src\pip\_internal\cli\main.py", line 9, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "d:\work\projects\pip\src\pip\_internal\cli\autocompletion.py", line 10, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "d:\work\projects\pip\src\pip\_internal\cli\main_parser.py", line 8, in <module>
    from pip._internal.cli import cmdoptions
  File "d:\work\projects\pip\src\pip\_internal\cli\cmdoptions.py", line 23, in <module>
    from pip._internal.cli.parser import ConfigOptionParser
  File "d:\work\projects\pip\src\pip\_internal\cli\parser.py", line 12, in <module>
    from pip._internal.configuration import Configuration, ConfigurationError
  File "d:\work\projects\pip\src\pip\_internal\configuration.py", line 21, in <module>
    from pip._internal.exceptions import (
  File "d:\work\projects\pip\src\pip\_internal\exceptions.py", line 7, in <module>
    from pip._vendor.pkg_resources import Distribution
  File "d:\work\projects\pip\src\pip\_vendor\pkg_resources\__init__.py", line 3252, in <module>
    def _initialize_master_working_set():
  File "d:\work\projects\pip\src\pip\_vendor\pkg_resources\__init__.py", line 3235, in _call_aside
    f(*args, **kwargs)
  File "d:\work\projects\pip\src\pip\_vendor\pkg_resources\__init__.py", line 3264, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "d:\work\projects\pip\src\pip\_vendor\pkg_resources\__init__.py", line 574, in _build_master
    ws = cls()
  File "d:\work\projects\pip\src\pip\_vendor\pkg_resources\__init__.py", line 567, in __init__
    self.add_entry(entry)
  File "d:\work\projects\pip\src\pip\_vendor\pkg_resources\__init__.py", line 623, in add_entry
    for dist in find_distributions(entry, True):
  File "d:\work\projects\pip\src\pip\_vendor\pkg_resources\__init__.py", line 2061, in find_on_path
    path_item_entries = _by_version_descending(filtered)
  File "d:\work\projects\pip\src\pip\_vendor\pkg_resources\__init__.py", line 2034, in _by_version_descending
    return sorted(names, key=_by_version, reverse=True)
  File "d:\work\projects\pip\src\pip\_vendor\pkg_resources\__init__.py", line 2032, in _by_version
    return [packaging.version.parse(part) for part in parts]
  File "d:\work\projects\pip\src\pip\_vendor\pkg_resources\__init__.py", line 2032, in <listcomp>
    return [packaging.version.parse(part) for part in parts]
  File "d:\work\projects\pip\src\pip\_vendor\packaging\version.py", line 59, in parse
    return LegacyVersion(version)
  File "d:\work\projects\pip\src\pip\_vendor\packaging\version.py", line 127, in __init__
    warnings.warn(
DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release

Sorry, that really is all I have time to do on this for now.

[pradyunsg]

pip._vendor.packaging.version.InvalidVersion: Invalid version: 'setuptools'

Whelp.

[pfmoore]

Yep. Setuptools is checking random bits of the filename to see if they are versions, relying on the version parser accepting anything without complaint. See here.

[pradyunsg]

I think we should do what you suggested in #9761 (comment), to unblock the CPython updates.

warnings.filterwarnings("ignore", category=DeprecationWarning, module=".*packaging\\.version")

[pfmoore]

Cool. I can't do this right now, as the pip test suite is failing all over the place on my machine. I've just spent ages tracking down an issue that makes virtualenv fail on my PC (fix incoming) and now I'm getting failures in the unit tests:

FAILED tests/unit/test_resolution_legacy_resolver.py::TestYankedWarning::test_sort_best_candidate__has_non_yanked - assert 2 == 0
FAILED tests/unit/test_resolution_legacy_resolver.py::TestYankedWarning::test_sort_best_candidate__all_yanked - assert 3 == 1
FAILED tests/unit/test_resolution_legacy_resolver.py::TestYankedWarning::test_sort_best_candidate__yanked_reason[-<none given>] - assert 3 == 1
FAILED tests/unit/test_resolution_legacy_resolver.py::TestYankedWarning::test_sort_best_candidate__yanked_reason[curly quote: \u2018-curly quote: \u2018] - assert 3 == 1

It looks like someone introduced some additional output, and mucked up testing somehow - one of the outputs that was expected to get one line is:

DEBUG    pip._internal.index.package_finder:package_finder.py:255 Given no hashes to check 1 links for project 'pkg': discarding no candidates
DEBUG    pip._internal.index.package_finder:package_finder.py:957 Using version 1.0 (newest of versions: 1.0)
WARNING  pip._internal.resolution.legacy.resolver:resolver.py:278 The candidate selected for download or install is a yanked version: 'mypackage' candidate (version 1.0 at https://example.com/pkg-1.0.tar.gz)

Those debug lines look suspicious 🙁

I think this actions run is the latest CI against main, but it seems to be broken. So I can't be sure if main is passing the tests in CI right now...

I've spent about as much time as I'm willing to for today tracking down test suite bugs and existing problems - so if anyone has a quick suggestion, please let me know. Otherwise, I'll get back to looking at adding that warning (and a test to check it does the job!) when things settle down, or I have more time/energy.

[pfmoore]

#9779 created for the "warnings as errors" problem, and #9780 for the test failures noted above.

[sbidoul]

I have done a review round on the issues and PRs in the 21.1 milestone.

If there are no other important / risky things people want to land in 21.1 (if there are, please let me know), I plan to do the release over the next week-end.

[sbidoul]

I plan to release tomorrow around mid day (Europe time).

[sbidoul]

I have the following highlights. Anything else @pypa/pip-committers want to add / remove ?

• A feature flag (--use-feature=in-tree-build) to enable in-tree build when
  installing or running pip wheel on local projects. This is intented to
  become the default in the future, so users are encouraged to test it and
  provide feedback.
• Improved UX of keyring integration.
• New resolver support of URLs in constraint files.
• Various UX and performance improvements in the new resolver.

[pradyunsg]

I'd call out the git security fix as well, using the phrasing from the NEWS fragment. And... A little bit of word-smithing below, which are mild suggestions. :)

I won't call it the "new resolver", just... dependency resolver. The older one is legacy after all. :P

• New resolver support of URLs in constraint files.

• Better handling of URLs in constraints files.

• Various UX and performance improvements in the new resolver.

• Performance and UX improvements to the dependency resolver.

[sbidoul]

And the release is done:

• wheel and sdist uploaded to pypi https://pypi.org/project/pip/21.1/
• pypa/get-pip updated Update to 21.1 get-pip#111
• announcement to discuss.python.org https://discuss.python.org/t/announcement-pip-21-1-has-now-been-released/8382
• CPython PR bpo-43930: Update bundled pip to 21.1 and setuptools to 56.0.0 python/cpython#25576

[sbidoul]

Incidentally this led me to the creation of my second bpo ever, the first one being... 229 months ago 😅

[pfmoore]

Apparently the PR needed approval to run CI. I've done that and I'll keep an eye on the results.

[sbidoul]

Apparently the PR needed approval to run CI.

That's a new GitHub policy to prevent crypto miners from hijacking GitHub actions. :/

I've done that and I'll keep an eye on the results.

Thanks!

[pfmoore]

That's a new GitHub policy to prevent crypto miners from hijacking GitHub actions. :/

Hmm, can I have a cut of the bitcoin you make, then? 🙂 💰

[pfmoore]

Also, I'll flag the PR for backport to 3.9 and 3.8. It should get into the final 3.8 release, 3.8.10, as a result.

[pradyunsg]

Looks like #9617 is gonna be the high traffic issue for this release, seemingly due to Ubuntu/Debian.

[pfmoore]

CPython PR merged to master. It looks like the backports failed, can you take a look @sbidoul? I've not got the time today (and possibly not this weekend) to look at this.

[pradyunsg]

Let's close the flood gates, and then we'll fix the smaller leaks. In other words, resolving the #9617 flood is fairly high priority IMO. If you're willing to make a 21.1.2 soon-ish, I'd strongly prefer that we release 21.1.1 based on what we have in main right now.

(heads up: I think the bugfix process in the docs might be slightly outdated)

As for waiting on packaging, I'll take a look at it later today, and see what the state of affairs is. It doesn't seem urgent or a release blocker in any way IMO. Same for #9841.

[sbidoul]

Ok. I'll do the release tonight, from main. If a new version of packaging is out by then I'll include it.
Regarding CPython I'm not sure. If that helps, I can do the PRs and let the CPython RM decide.

[sbidoul]

I think the bugfix process in the docs might be slightly outdated

Yep I had noticed that. In practice it should be identical to the main release process except we branch off the previous tag (when we can't use main). Or is there something I overlooked ?

[pfmoore]

Regarding CPython I'm not sure. If that helps, I can do the PRs and let the CPython RM decide.

OK. Make sure you ping the RMs for approval on the PRs, I won't merge them myself unless they are approved.

[pradyunsg]

Or is there something I overlooked ?

No, you're correct. What I've done in the past is roughly:

• git checkout last-release
• git branch release/YY.M.m
• git cherry-pick -m 1 hash, for each merged PR we want
• (all the regular nox + file-a-PR stuff, that we'd do for releases)

That said, you can probably get away with just cutting it off-of-main, since we wanna include everything.

[puetzk]

my view is that we should wait for a packaging release, and include that into pip following our normal vendoring process. If this has been broken since CPython 3.7, it doesn't seem like it's a sufficiently urgent fix to warrant an exception to our normal policy of waiting for a downstream fix.

Broken since 3.8 (3.7 works), but yeah, I do have to admit that's a defensible position. I'm surprised there aren't more people getting burned by this (I would have thought calling python in MSVC tooling was more common), but apparently not: neither #8649 nor BPO38989 attracted a large number of "me too" commenters.

So while it would be nice to hit the last 3.8 patch and fix this 3.7->3.8 regression before the branch dies (then all 3.x would work), my stronger interest is more on getting the fix into some working build that is still supported, since 3.7 is now into "security only, and no binary releases", so I'd like to be able to take a newer version. If we miss Monday's CPython but a pip 21.1.2 can happen that will get into 3.9.6, and 3.10.0b2 (3.8 just stays broken, as it has been its entire life), that's enough.

[pfmoore]

3.8 just stays broken, as it has been its entire life

That seems a bit strong, surely it's simply a matter of upgrading pip once there's a fixed version, so 3.8 will hardly "stay broken".

[puetzk]

Good point, pip itself is a py3-none-any.whl, so there isn't a wrong-architecture wheel for it to pick and the bug won't manifest. So you're right; one could upgrade your way out of it with upgrade-ensurepip once there was a newer pip available, even if the fix missed 3.8.10

[pfmoore]

I wouldn't recommend hacking the stdlib module. Just upgrade pip in the environment you work in (python -m pip instyall -U pip). If you use virtualenv, environments get created with the latest available pip anyway.

[pradyunsg]

Or, like, run python -m ensurepip --upgrade when you wanna install pip?

[sbidoul]

21.1.1 uploaded to PyPI

• wheel and sdist uploaded to pypi https://pypi.org/project/pip/21.1.1/
• pypa/get-pip updated https://github.com/pypa/get-pip/tree/21.1.1
• announcement to discuss.python.org
• CPython PR bpo-43993: Update vendored pip to 21.1.1 python/cpython#25761

[pradyunsg]

21.1.1 has had the intended effect. I'm no longer getting ridiculous amounts of reputation everyday on StackOverflow. 🤣

[sbidoul]

It looks like things are getting quieter. Shall we do a last 21.1 bugfix release soon ? Milestone 21.1.2 is populated with what has been identified so far. Is there anything else that needs to land in it ?

[puetzk]

That would be the last release that could get into python 3.10, right? I see right now that 21.2 milestone says "Due by July 31, 2021", which seems too late vs PEP 619 (it would fall between beta 4 and candidate 1, and I would think CPython doesn't want a new minor version in the release candidate).

If so I would think we whatever pip gets bundled with 3.10 needs pypa/packaging#396 (more than 3.8/3.9 did), to avoid getting hit by the new DeprecationWarning (per PEP 632 "import distutils will raise a deprecation warning" in 3.10 and 3.11, then be removed in 3.12). Currently packaging.tags v20.9 still uses distutils. Unless @pradyunsg needs a fresh source of warnings to mine for stackoverflow upvotes :-)

And yes, it would also fix #8649/#9962, which I would really selfishly like so we could upgrade past python 3.7 without builds picking up wrong-architecture wheels all the time :-)

[uranusjr]

Not sure about the policy from beta to rc, but CPython upgrades the bundled pip between minor releases, so even if we can’t get 21.2 into 3.10.0 we can still have it in 3.10.1.

[ambv]

During beta you can still freely upgrade pip for 3.10.0. We'd rather not do that during the RC phase since this is supposed to be a final stage when we stabilize the "point-oh" release which is always the most stressful and crucial.

[sbidoul]

I just uploaded 21.1.2 to PyPI

• wheel and sdist uploaded to pypi https://pypi.org/project/pip/21.1.2/
• pypa/get-pip updated https://github.com/pypa/get-pip/tree/21.1.2
• announcement to discuss.python.org
• CPython PR

[sbidoul]

I realize I forgot to do 21.1.3. There are currently two issues in that milestone.
I'll do that in the coming days. That will be the last one in 21.1.

Let me know if there are other things to include.

[pradyunsg]

Fun fact: 21.1.3 will be the 100th tag on this repository!

[pradyunsg]

@sbidoul If you could include 8b521e2, that'd be great! I'll add the corresponding PR to the milestone.

[sbidoul]

I just uploaded 21.1.3 to PyPI

• wheel and sdist uploaded to pypi https://pypi.org/project/pip/21.1.3/
• pypa/get-pip updated https://github.com/pypa/get-pip/tree/21.1.3
• announcement to discuss.python.org
• CPython PR bpo-44516: update vendored pip to 21.1.3 python/cpython#26912

[sbidoul]

@pradyunsg all automations worked perfectly and the release process documentation is perfect as well, thank you !