Pre releases?

[nicoddemus]

I was wondering if we should experiment with making official pre-releases, before the official major release is out. While I understand most people don't really test against pre-releases, pytest is used largely enough that some parties might be interested to test pre-releases in their code bases and be early reporters.

Thoughts?

[gnikonorov]

+1 on this. I think it’s a great idea

[bluetech]

+1 from me too.

Are you thinking more of a "beta" (=> some changes still allowed) or "release candidate" (=> pre-releases made after branching from master, only backported bugfixes allowed)?

[nicoddemus]

Are you thinking more of a "beta" (=> some changes still allowed)

That's what I had in mind, more of a "let's test the waters before the release" rather than "lets feature freeze master", but I'm of course open for suggestions. 😁

[gnikonorov]

That's what I had in mind, more of a "let's test the waters before the release" rather than "lets feature freeze master", but I'm of course open for suggestions. 😁

+1 for the "let's test the waters before the release" approach. What schedule were you thinking for beta releases, @nicoddemus ?

[nicoddemus]

I was thinking along the lines of releasing a beta when we are "done" with all we had set for the release milestone, and see how that goes. Then we might make new releases around 1 week a part if there are regressions, and make the final release if no more regressions come up.

This is all a ballpark of course, the time between releases might be shorter or longer depending on our availability.

[gnikonorov]

I like the idea. Definitely +1 from me

[nicoddemus]

OK, I plan to make a release candidate tomorrow then.

I'll leave this open so we can evaluate how this will pan out once 6.0 is out, with two outcomes:

1. We have a few release candidates with many reports, making the 6.0 release smoother. Success!
2. Nobody reports anything for the first release candidate, so we release 6.0 and a ton of regressions show up. So the experiment failed.

So let's leave this open to update the docs if 1 happens, or just close this with a sad face on 2. 😁

[asottile]

I'm down for trying this out, it does add a bit of work to releasing (and in practice, sadly very few utilized the pre-releases I did for flake8, pycodestyle -- most of the release bugs were found after final).

[The-Compiler]

I gave 6.0.0rc1 a try with qutebrowser and ran into an issue, not sure if pylint or pytest is to blame though 😅: #7473.

I thought I had found another issue related to logging, but it's qutebrowser rather than pytest to blame for that one: qutebrowser/qutebrowser@e206d34 (it's #7231 which made it show up, it was there before).

I'd also like to have some more automation in qutebrowser's CI to automatically test prereleases (or perhaps even git master branches) of dependencies I use, see qutebrowser/qutebrowser#5582.

[symonk]

I think this ticket has been a great success thus far, I've been very busy for a while but it seems the current rc is flagging a few serious issues with some users who chose to roll up to try it out. Getting visibility of these issue(s) is great as early as possible.

[nicoddemus]

I think this ticket has been a great success thus far,

I agree! After 6.0 is out, I plan to write an workflow for future pre-releases.

[hugovk]

Curious about the process:

When 6.0.0 final comes out, will it be from master, or from 6.0.0rc1 plus cherry picks?

[nicoddemus]

Hi @hugovk, sorry for the delay.

I think it will come from master directly, I don't think there's anything which is on master at the moment that wouldn't be ported to 6.0.0 anyway.

But for the next releases probably it makes sense to follow normal protocol and create a 7.0.x at the time 7.0.0rc1 gets released?

[The-Compiler]

I think this was a great success, especially given that @hroncok rebuilt all Fedora packages and went through the failing logs - thanks a lot again for that, there was a lot we could fix that way before things broke apart for everyone.

[nicoddemus]

Agree on all fronts. 😁

Let's leave this open, I plan to close this once I open a PR for docs about the pre-release process. 👍

[hroncok]

One feedback: I would prefer if the "error on deprecated stuff" switch happened at rc, not at final. It basically changes the behavior and stuff that "worked" with rc1 no longer "work" with final. Ideally, rc would behave like final in all ways predictable. WDYT?

[nicoddemus]

One feedback: I would prefer if the "error on deprecated stuff" switch happened at rc, not at final. It basically changes the behavior and stuff that "worked" with rc1 no longer "work" with final. Ideally, rc would behave like final in all ways predictable. WDYT?

Definitely agree! That really was a slip up on my part.

The process described by the new docs about the prereleases should prevent that from happening in the future, because the maintenance branch will be created immediately before the first prerelease, so any change needs to be backported from now on, and only bugfixes are included in backports by definition.

[hroncok]

Glad to hear that there is an agreement on that part for future releases. Thanks.

[nicoddemus]

Interesting information I just came across:

Here's a graph of downloads of the 6.0.0rc1 release vs 6.0.1 courtesy of pypy.tech:

Also some totals as of today:

So we actually had a good number of downloads for the release candidate (of course there's a number of automated downloads/mirrors there, but still).

[hugovk]

The pytest-rerunfailures plugin has failures with the removals in 6.1.0 but was evidently fine with 6.0.0: pytest-dev/pytest-rerunfailures#128 (affecting at least pip, Matplotlib and twine).

It's probably a good idea to also do pre-releases with the x.1.0 versions that actually remove the deprecations, in addition to the x.0.0 major bump (re: Backwards Compatibility Policy).