Revamp go-ipfs Release Flow #6482
Conversation
docs/releases.md
Outdated
|
|
||
| # 🏗 API Changes | ||
|
|
||
| <List of API changes, if any> |
There was a problem hiding this comment.
The 🔦 Highlights and 🏗 API Changes could potentially be replaced by the Changelog. That said, there is still value on highlighting some of the changes, the ones that we want users to know about first (or the ones that we want to ensure that users see, in case they don't have time to read the entire changelog).
There was a problem hiding this comment.
Or, more generally, breaking changes/notices.
docs/releases.md
Outdated
| - [ ] all tests pass (no exceptions) | ||
| - [ ] run interop tests https://github.com/ipfs/interop#test-with-a-non-yet-released-version-of-go-ipfs | ||
| - [ ] webui works (for most definitions of 'works') - Test the multiple pages and verify that no visible errors are shown. | ||
| ### Release Stage 1 - Internal testing |
There was a problem hiding this comment.
Note: as noted on Release Flow, this whole stage can potentially be automated. Until it is, let's have the items as a check list so that we don't forget any :)
docs/releases.md
Outdated
| - [ ] Open Bazaar | ||
| - [ ] QRI | ||
| - [ ] Run tests available in the following repos with the latest RC (check when all tests pass): | ||
| - [ ] [orbit-db](https://github.com/orbitdb/orbit-db) |
There was a problem hiding this comment.
I'm pretty sure that both of the lists here can be way bigger. We probably should have a "expectations description" before adding more folks
There was a problem hiding this comment.
Can we test this against go-ipfs? I thought it embedded a js-ipfs node.
There was a problem hiding this comment.
AFAIK, it can do both. Will check again.
For the other readers. What @Stebalien meant by this was just referring to orbit-db and not all the other apps.
docs/releases.md
Outdated
|
|
||
| ### Release Stage 4 - Complete the Release | ||
|
|
||
| - Take a snapshot between of everyone that has contributed to this release (including its subdeps in IPFS, libp2p, IPLD and multiformats) using [`name-your-contributors`](https://www.npmjs.com/package/name-your-contributors). Generate a nice markdown list with [this script](https://gist.github.com/alanshaw/5a2d9465c5a05b201d949551bdb1fcc3). Add it to this issue. |
There was a problem hiding this comment.
This gist needs to be updated to point to the go code repos. Thanks to @alanshaw for creating it in the first place.
There was a problem hiding this comment.
The bin/mkreleaselog script currently does this. Output: https://blog.ipfs.io/93-go-ipfs-0.4.21#huge-thank-you-to-everyone-that-made-this-release-possible
|
@Stebalien let me know what you think of this Release Flow. I would be happy to coordinate the first cycle if you would like :) |
Co-Authored-By: Michael Burns <5170+mburns@users.noreply.github.com>
docs/releases.md
Outdated
|
|
||
| - **Release Stage 1 - Internal testing** - Test the Release against our testing infrastructure, including interoperability, integration, test lab, multiple runtimes and the apps we've built (WebUI, Desktop, NPM on IPFS, HTTP Client Libraries). The intent is to make this stage fully automated (and somewhat is already), until then, we manually check a list and ensure that all tests have been run | ||
| - **Release Stage 2 - Invite IPFS partners to test** - Reach out to our partners (i.e. projects that have volunteered to support `go-ipfs` by using their own test infrastructure and tell us the results) | ||
| - **Release Stage 3 - Announce to the broader community** - Communicate to the community that a new Release Candidate is ready and that everyone is welcome to test it with us |
There was a problem hiding this comment.
Any real reason not to combine stages 2 and 3? Really, we usually ask the broader community (stage 3) first because they tend not to be using IPFS for mission critical operations. That is:
- Advanced/eager users tend to find "shallow" bugs: api breakage, bad documentation, etc.
- Partners tend to find deep bugs: performance regressions, rare crashes, etc.
There was a problem hiding this comment.
My rationale for this separation was based on my assumption that we are very limited in capacity to deal with incoming bug reports.
I expect that our early testers are more knowledgeable and therefore, more capable of debugging themselves part of the problem or at least providing more useful information, which will save us a ton of time and cover a ton of bugs.
In contrast, I expect that the broad community to be able to also pick on bugs but provide less useful information. By doing early testers first, we save some time from debugging less documented/debugged bugs (which would overlap with the ones that our early testers will report).
There was a problem hiding this comment.
That said, I'm not holding any strong feelings towards the flow. Happy to take in the wisdom of the maintainers and iterate on it :)
There was a problem hiding this comment.
What about:
- Stage 2 - Announce a preliminary RC on IRC and invite the community to run sanity checks.
- Stage 3 - Pull in partners.
- Stage 4 - Announce the RC on twitter etc? I wonder if there's some way to allow, e.g., ipfs-desktop users to opt into this release.
I guess I mostly just want a chance to test it with lower stakes than people relying on IPFS in production systems.
My rationale for this separation was based on my assumption that we are very limited in capacity to deal with incoming bug reports.
Most of the bug reports we get from announcements on IRC are actually pretty good. They also tend to be pretty straight forward to diagnose as they aren't subtle performance issues.
There was a problem hiding this comment.
lower stakes than people relying on IPFS in production systems.
We should not advise folks to run RC in production, rather just invite them to test on their test infra and QA if they have it built.
Most of the bug reports we get from announcements on IRC are actually pretty good. They also tend to be pretty straight forward to diagnose as they aren't subtle performance issues.
This is good info to know. If you feel confident on merging Stage 2 and 3, then let's do it. Just confirm and I'll update it
There was a problem hiding this comment.
It's sounds weird that certain releases are meant for partners. It sounds like giving early access to features even though it means "try it out". It should be very clear that anyone can try it out. I can also imagine that different partners (and users) may choose to test ipfs pre-releasees at different stages depending on circumstances.
Note that it is also weird that we give some partners a half-baked release to test. Either they test it by deploying it to their users, in which case they should not be given a half-baked release that can break their apps (even if they volunteer), or they have they're own testing environments, in which case they could directly master or a release branch continuously without waiting for a specific release stage.
Otherwise said:
- Sounds good to have alpha/beta/rcs (we should have a release branch that includes all features in the release and only updates with bugfixes). We should be able to skip certain checkpoints sometimes (like hotfixes to specific issues).
- Document what the level of maturity is after each checkpoint, so that everyone can test and try it out (partner or not).
- Create a known and consistent venue to gather feedback on each release (i.e. an issue) and include information from our own testing in it.
- Let people choose when they test based on their availability, feedback on the current state of the release and issues with the previous release.
There was a problem hiding this comment.
We absolutely must not encourage partners to run a RC in production that can potentially harm their service and hurt their users. What we hope is for partners to use their own Beta channels, Testing Infrastructure and potentially QA setup to let us know if the new release passes the tests.
There was a problem hiding this comment.
We absolutely must not encourage partners to run a RC in production that can potentially harm their service and hurt their users.
Any release we make can potentially break something. This is an inviolate rule of software and we must not lie to ourselves about this. The goal here is to minimize the impact by deploying in stages, not to avoid all harm.
We definitely shouldn't ask our partners to run RCs that haven't been tested but we should absolutely ask them to run an RC that we believe to be "ready for release", to reduce the impact of any bugs they may encounter. That way they can roll back, we can fix the issue, and then move on with the release cycle without waiting for another release.
What we hope is for partners to use their own Beta channels, Testing Infrastructure and potentially QA setup to let us know if the new release passes the tests.
This would be awesome but it's asking a lot from our partners and isn't currently the weakest part of our release process.
There was a problem hiding this comment.
You know, we probably can just call these alpha, beta, etc. That should make this easier to explain to users:
- Dev - Bleeding Edge.
- Alpha - Feature Freeze.
- Beta - Tested but may still be buggy.
- RC1 - Release quality, small rollout.
- RC2 - Release quality, larger rollout.
- Release - Full rollout.
docs/releases.md
Outdated
|
|
||
| # 🏗 API Changes | ||
|
|
||
| <List of API changes, if any> |
There was a problem hiding this comment.
Or, more generally, breaking changes/notices.
docs/releases.md
Outdated
| - [ ] sharness | ||
| - [ ] [interop](https://github.com/ipfs/interop#test-with-a-non-yet-released-version-of-go-ipfs) | ||
| - [ ] Network Testing: | ||
| - [ ] test lab things |
There was a problem hiding this comment.
@bigs isn't there a thing?
There was a problem hiding this comment.
It is a thing but not yet for IPFS just yet.
docs/releases.md
Outdated
| - [ ] Network Testing: | ||
| - [ ] test lab things | ||
| - [ ] Infrastructure Testing: | ||
| - [ ] Deploy new version to a subset of Bootstrappers |
There was a problem hiding this comment.
Can we even do this yet? cc @ipfs/wg-infrastructure.
docs/releases.md
Outdated
|
|
||
| ### Release Stage 4 - Complete the Release | ||
|
|
||
| - Take a snapshot between of everyone that has contributed to this release (including its subdeps in IPFS, libp2p, IPLD and multiformats) using [`name-your-contributors`](https://www.npmjs.com/package/name-your-contributors). Generate a nice markdown list with [this script](https://gist.github.com/alanshaw/5a2d9465c5a05b201d949551bdb1fcc3). Add it to this issue. |
There was a problem hiding this comment.
The bin/mkreleaselog script currently does this. Output: https://blog.ipfs.io/93-go-ipfs-0.4.21#huge-thank-you-to-everyone-that-made-this-release-possible
docs/releases.md
Outdated
| Would you like to contribute to the IPFS project and don't know how? Well, there are a few places you can get started: | ||
|
|
||
| - Check the issues with the `help wanted` label in the [go-ipfs repo](https://github.com/ipfs/go-ipfs/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22) | ||
| - Join an IPFS All Hands, introduce yourself and let us know where you would like to contribute - https://github.com/ipfs/team-mgmt/#weekly-ipfs-all-hands |
There was a problem hiding this comment.
Maybe we should start pointing people at IRC/Matrix instead of the All Hands.
There was a problem hiding this comment.
Multiple people have reported that showing up on the main channel on IRC with ideas and questions typically results into them falling into oblivion.
If someone is new, it is nice to see that there are humans, that is why the call is important.
docs/releases.md
Outdated
| - Join an IPFS All Hands, introduce yourself and let us know where you would like to contribute - https://github.com/ipfs/team-mgmt/#weekly-ipfs-all-hands | ||
| - Hack with IPFS and show us what you made! The All Hands call is also the perfect venue for demos, join in and show us what you built | ||
| - Join the discussion at http://discuss.ipfs.io/ and help users finding their answers. | ||
| - Join the [Go Core Dev Team Weekly Sync](https://github.com/ipfs/team-mgmt/issues/650) and be part of the Sprint action! |
There was a problem hiding this comment.
Is there a replacement?
|
|
||
| Post `go-ipfs` 1.X.X (future), `go-ipfs` will use semver. This means that only major releases will contain breaking changes, minors will be reserved for new features and patches for bug fixes. | ||
|
|
||
| We do not yet retroactively apply fixes to older releases (no Long Term Support releases for now), which means that we always recommend users to update to the latest, whenever possible. |
There was a problem hiding this comment.
Ho, bummer...
Please consider this in the future for disruptive problems. At Infura we have been stuck with problematic release for months without solution (other than forking): 0.4.19 and 0.4.20 had memory usage problem but we couldn't revert to 0.4.18 due to security reason. This will only become more critical in the future.
There was a problem hiding this comment.
In that case, we would have applied the CPU fixes in 0.4.19 to 0.4.18 as a patch. This is just saying that we're not going to release a 0.4.20 and then go back and patch a 0.4.18 release.
|
Hi folks! Seems that we are arriving at good port. I was hoping to have this reviewed and approved by everyone so that we can use it for #6484. @Stebalien @momack2 is this a goal that you agree that we should be shooting for? |
docs/releases.md
Outdated
| - **Release Stage 3 - Announce to the broader community** - Communicate to the community that a new Release Candidate is ready and that everyone is welcome to test it with us | ||
| - **Release Stage 4 - Complete the Release** - Finalize the release, start the next release. | ||
|
|
||
| The Release Stages are not linked to Release Candidate numbers, in fact, there can be multiple release candidate per stages as we catch bugs and improve the release itself. |
There was a problem hiding this comment.
Are we going to be making this clear via some form of numbering of the rc, i.e. v0.4.22-rc-1-3 => Stage 1, iteration 3?
There was a problem hiding this comment.
The plan is just to use one number. The RC- signals that there has been a new one published
There was a problem hiding this comment.
See my comment on alpha, beta, etc. above.
docs/releases.md
Outdated
| - [ ] sharness | ||
| - [ ] [interop](https://github.com/ipfs/interop#test-with-a-non-yet-released-version-of-go-ipfs) | ||
| - [ ] Network Testing: | ||
| - [ ] test lab things |
There was a problem hiding this comment.
It is a thing but not yet for IPFS just yet.
docs/releases.md
Outdated
| - [ ] IPFS HTTP Client Libraries Testing: | ||
| - [ ] [JS](http://github.com/ipfs/js-ipfs-http-client) | ||
| - [ ] [Go](https://github.com/ipfs/go-ipfs-api) | ||
| - [ ] IPFS Application Testing - Run the tests of the following applications: |
There was a problem hiding this comment.
We're going to need DRIs for this. Maybe add a line about pinging the relevant maintainers?
docs/EARLY_TESTERS.md
Outdated
| ## Who has signed up? | ||
|
|
||
| - Infura | ||
| - Textile |
There was a problem hiding this comment.
Let's add contacts here (i.e., github handles).
There was a problem hiding this comment.
@MichaelMure @sanderpick - are you folks on board with helping us test out new releases?
There was a problem hiding this comment.
As of right now, Infura doesn't have a way to do canary testing, but we are aiming for it. Please send us notification and we'll get to proper testing eventually.
There was a problem hiding this comment.
We don’t have an automated way to try different releases but we can certainly try all the candidates as they come out.
There was a problem hiding this comment.
Having RTrade/Temporal added to the list would be helpful as well and save us some trouble :)
docs/releases.md
Outdated
|
|
||
| - **Release Stage 1 - Internal testing** - Test the Release against our testing infrastructure, including interoperability, integration, test lab, multiple runtimes and the apps we've built (WebUI, Desktop, NPM on IPFS, HTTP Client Libraries). The intent is to make this stage fully automated (and somewhat is already), until then, we manually check a list and ensure that all tests have been run | ||
| - **Release Stage 2 - Invite IPFS partners to test** - Reach out to our partners (i.e. projects that have volunteered to support `go-ipfs` by using their own test infrastructure and tell us the results) | ||
| - **Release Stage 3 - Announce to the broader community** - Communicate to the community that a new Release Candidate is ready and that everyone is welcome to test it with us |
There was a problem hiding this comment.
We should not advise folks to run RC in production, rather just invite them to test on their test infra and QA if they have it built.
We need to run our final RC in production before releasing it. Basically, we have two options:
- We cut a release with a performance regression an wait a release cycle (or go back through the release cycle for a patch release).
- We cut release candidate and ask partners to start running it. This way, we can fix any performance regressions without starting a new release cycle.
This RC should be a true RC, a candidate for release.
I believe this comes down to an issue of naming. Really, the stages are:
- Alpha - Internal: Small sample size, light and heavy workloads.
- Beta - IRC: Medium sample size, light workloads.
- RC1 - Partners: Medium sample size, heavy workloads.
- RC2 - End users (twitter): Large sample size.
However, IPFS itself is "alpha" so this would be even more confusing.
I'd love to get partners on board at the "beta" stage but I'm not sure how helpful that will be unless they're willing to build out simulations for us. Having more testing on light workloads won't help all that much.
docs/releases.md
Outdated
| - **Release Stage 3 - Announce to the broader community** - Communicate to the community that a new Release Candidate is ready and that everyone is welcome to test it with us | ||
| - **Release Stage 4 - Complete the Release** - Finalize the release, start the next release. | ||
|
|
||
| The Release Stages are not linked to Release Candidate numbers, in fact, there can be multiple release candidate per stages as we catch bugs and improve the release itself. |
There was a problem hiding this comment.
See my comment on alpha, beta, etc. above.
docs/releases.md
Outdated
| - [ ] IPFS HTTP Client Libraries Testing: | ||
| - [ ] [JS](http://github.com/ipfs/js-ipfs-http-client) | ||
| - [ ] [Go](https://github.com/ipfs/go-ipfs-api) | ||
| - [ ] IPFS Application Testing - Run the tests of the following applications: |
There was a problem hiding this comment.
We're going to need DRIs for this. Maybe add a line about pinging the relevant maintainers?
docs/releases.md
Outdated
| Would you like to contribute to the IPFS project and don't know how? Well, there are a few places you can get started: | ||
|
|
||
| - Check the issues with the `help wanted` label in the [go-ipfs repo](https://github.com/ipfs/go-ipfs/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22) | ||
| - Join an IPFS All Hands, introduce yourself and let us know where you would like to contribute - https://github.com/ipfs/team-mgmt/#weekly-ipfs-all-hands |
docs/releases.md
Outdated
|
|
||
| - [ ] Documentation | ||
| - [ ] Ensure that CHANGELOG.md is up to date | ||
| - [ ] Ensure that README.md is up to date |
There was a problem hiding this comment.
I past we had times when parts of docs/config.md and docs/experimental-features.md got out of date. Its the source of truth for many power users. It should be a part of the checklist:
| - [ ] Ensure that README.md is up to date | |
| - [ ] Ensure that README.md is up to date | |
| - [ ] Ensure that `docs/config.md` and `docs/experimental-features.md` are up to date |
Co-Authored-By: Marcin Rataj <lidel@lidel.org>
|
Meeting notes from our sync today: https://github.com/ipfs/team-mgmt/pull/994/files |
And other nits.
|
|
||
| ## Changelog | ||
|
|
||
| < changelog generated by bin/mkreleaselog > |
There was a problem hiding this comment.
This is going to be really long: https://github.com/ipfs/go-ipfs/blob/master/CHANGELOG.md#changelog
There was a problem hiding this comment.
Seems that most of it is libp2p releated. Shouldn't that just appear on go-libp2p releases?
There was a problem hiding this comment.
It's about half libp2p. Once libp2p starts making releases with changelogs, we can just include a link to the libp2p release and leave it at that.
But that's still over 100 lines. Let's leave it for now but we may need to hide this behind a <details>.
docs/RELEASE_ISSUE_TEMPLATE.md
Outdated
| - version string in `version.go` has been updated | ||
| - tag commit with vX.Y.Z-rcN | ||
| - [ ] Stage 1 - Internal Testing | ||
| - [ ] Feature freeze. If any non-trivial features get added to the release, uncheck all the checkboxes and return to this stage. |
There was a problem hiding this comment.
Maybe "non-trivial non-bugfix changes"? I'm thinking about things like minor performance optimizations.
docs/releases.md
Outdated
| Post `go-ipfs` 1.X.X (future), `go-ipfs` will use semver. This means that only major releases will contain breaking changes, minors will be reserved for new features and patches for bug fixes. | ||
|
|
||
| We do not yet retroactively apply fixes to older releases (no Long Term Support releases for now), which means that we always recommend users to update to the latest, whenever possible. | ||
| TBW |
Reform release distribution checklist + add chocolatey package
| - [ ] Textile (@sanderpick) | ||
| - [ ] Pinata (@obo20) | ||
| - [ ] RTrade (@postables) | ||
| - [ ] QRI (@b5) |
There was a problem hiding this comment.
@Stebalien did you get a confirmation from all of these after getting this PR up? It would be good to double check given that we only have now a description of what it is.
Hi folks! As discussed yesterday, here is a first stab at revamping the Release Flow. Welcome for feedback :)