Release resolver-centric beta of pip in early July

[brainwane]

Should we publish a release of pip in May that has a beta of the resolver?

A lot more functionality has gone in since the alpha resolver release in pip 20.1 -- we may well publish a pip 20.1.1 in May that has those improvements in a second alpha. And, functionality-wise, @pradyunsg and @pfmoore and @ilanschnell believe we are in good shape for a beta.*

But we need to learn more about how much the existing error messaging and user experience in general is a stumbling block for users. Because, as I see it, that's what determines which of these we do:

a) If it's a big problem, then we should not do the beta release and publicity work, because we'll just get that feedback back (and not as much testing of functionality), and it'll be hard for less expert users to test it. And our bug triagers, packaging colleagues, and other helpers will have to deal with the increased support load but without as much benefit in useful testing outcomes. So, in this case, we should delay the beta till we can do more work on that, and we should have our user experience researchers use their own recruiting methods to survey/interview/test users to get data on the error message situation.

b) If the error messaging is good enough for users to use for rough troubleshooting, then maybe we should go ahead and do the beta in May, then ask users for feedback so we can iterate on that ("tell us how you want this to be presented") as part of the beta cycle, and use pip's/PyPA's existing communication networks and recruiting methods to get beta testers.

So this is a question that @nlhkabu and @ei8fdb (cc @georgiamoon) need to help us decide, preferably by sometime Monday the 18th (to give us time to prep the testing plan, etc.). Am starting this issue as a place for us to decide this.

(Related to #7628, #6536, #988.)

---

* "A beta" here means:

• release a version whose version number, per PEP 440, denotes that it's a beta prerelease
• make a manual test plan that has specific areas, workflows, OSes/versions/environments, etc. that we want people to play with, e.g., in their CI setups
• publicize the beta via various announcement channels/media and encourage bug reports, working especially to publicize it to people who don't usually pay attention to Python packaging stuff (see Oh look! It's April again! (Release 20.1) #7951 (comment) and subsequent comments for examples)

[pradyunsg]

we may well publish a pip 20.1.1 in May that has those improvements in a second alpha.

I think we shouldn't have these changes as part of pip 20.1.1, which should only contain specific bugfixes on top of pip 20.1. Instead, I'd prefer we call it pip 20.2b0, which we can cut directly from master, while including all the other post-20.1 changes.

That also unties us from any ongoing discussions/details that might be hold up a pip 20.1.1 release. :)

[brainwane]

OK, so, we decided as a group that we'll be releasing a beta in June. Things that deserve their own issue will go in the "To do before beta" column in the Resolver Rollout board but little comments, updates, TODOs, etc. can go here.

[brainwane]

I've just looked in the Changelog while working on a "here's what's new in the beta, here's what to test" writeup. Other than the resolver what are the major things to highlight?

[brainwane]

@xavfernandez @chrahunt @cjerdonek @sbidoul - are there any other changes in pip that you are aiming to land between now and late July? Right now I'm assuming the only significant new feature or fix since 20.1 is the new dependency resolver, but I I certainly don't want to drown out other news. (meeting notes about this)

[sbidoul]

@brainwane on my side I'd like to land

• PEP 376 REQUESTED (Implement PEP 376 REQUESTED #8026, ready to merge), which I believe will enable interesting use cases

Then if time allows,

• the deprecation of the setup.py install code path, to gather feedback on how important it still is (Raise a warning when pip falls back to the "legacy" direct install code #8102, awaiting the decision on the feature flag mechanism to move forward)
• if there is enough mental energy around for this, control out-of-tree builds with a feature flag (Solving issues related to out-of-tree builds #7555), but we'd need to come to a definitive decision about build-via-sdist first

[pfmoore]

+1 on all of the features @sbidoul noted. I'd say just go ahead and merge #8026, it doesn't look like it's blocked on anything.

[brainwane]

See the "To Do Before Beta" column in https://github.com/pypa/pip/projects/6 and the similar column in https://github.com/pypa/pip/projects/5 to see what we need to finish before we can declare beta.

[brainwane]

We decided not to release a resolver-centric beta this month. However, pip's release manager is free to decide whether to release a pip beta to get testing for other features, such as "Parallelize pip list --outdated and --uptodate" #8504.

The full meeting notes are up but, in summary: most of the contractor team is not yet comfortable giving users the new resolver by default, and would not be comfortable doing so in July even after a three-week or four-week beta cycle. Therefore, we will widely publicize pip 20.2 and tell people that, just as 20.1 included an alpha of the new resolver, 20.2 includes a beta of the new resolver. Then, in late July, August, and September, we'll recruit testers, make pre-releases as appropriate, and get and use as much feedback as possible. We aim to then release pip 20.3 in October 2020 with the new resolver on by default. See #8512 and #8371 for a bit more on that.

I am closing this issue since it was specifically about a resolver-centric beta, but the release manager can open a new issue if they'd like to release 20.2b2 concentrating on other features.