ci: adjust the release process to handle publishing ops and ops-scenario

[tonyandrewmeyer]

Adds new publish workflows for ops-scenario (one for main PyPI, one for test.pypi.org), modelled on the ops workflows. These differ significantly from the old ops-scenario publish workflow (now deleted), which would automatically create a GitHub release and tag and publish to PyPI (with API credentials) on a push to main.

To avoid confusion, the publish workflows for ops are renamed (and while this is being done, the extension is changed from yml to yaml to match all the other files).

The publish workflows are trigged by new tags - 2.* for ops (rather than a release being published) and scenario-7.* for ops-scenario. This allows releases that only include publishing one of the packages (for example, in a month that had no changes to one, or when a patch release is required).

The HACKING.md documentation is updated to reflect the changes in the release process.

A CHANGES.md is added for Scenario, backdated to 7.0.0 (although 7.0.0 is not entirely complete, but it does list the main changes).

Once this is merged, trusted publishing will need to be configured for the ops-scenario package, and the config for ops will need to be adjusted to refer to the new workflow filename.

Drive-bys:

• The smoke test workflow had incorrect indentation for the schedule trigger.
• Clarify when (testing) and (harness) scopes are used in conventional commits, now that we have Scenario in the repo.
• Note the use of the various .. juju directives in documentation.
• Minor clarifications in the release process based on comments from @james-garner-canonical when he did his first release.

[james-garner-canonical]

Thanks Tony, looking good. Just a few more questions from me, sorry!

[james-garner-canonical]

Worth adding a note about choosing the previous release tag to compare to?

I think the rule is: find the previous release tag for each thing you're releasing, and if there is more than one (i.e. an ops only release and a scenario only release), use the older of the two.

Say you had an ops release, then a scenario release, and are now doing a joint release -- you'd want to choose the ops release to generate release notes.

But you'd need to filter out any scenario PRs from before the last scenario release, since they'd have been covered already.

[tonyandrewmeyer]

Worth adding a note about choosing the previous release tag to compare to?

I think the rule is: find the previous release tag for each thing you're releasing, and if there is more than one (i.e. an ops only release and a scenario only release), use the older of the two.

Say you had an ops release, then a scenario release, and are now doing a joint release -- you'd want to choose the ops release to generate release notes.

But you'd need to filter out any scenario PRs from before the last scenario release, since they'd have been covered already.

Yes, this is another place where it gets messy, at least in theory. I think in practice it will be unusual to release only one of the packages, so it ends up being fairly simple. The rule is indeed to pick the oldest I think.

The instructions also don't cover what to do for a patch release, or when doing a normal release after a patch release. I'm leaving that for someone else to explain :)

[james-garner-canonical]

We can add it to the release instructions! "If you're the first one to do a patch release, update the release instructions with what you did and remove this directive."

[james-garner-canonical]

Clarification: mostly joking. I'll hit approve now as I don't think any of this is worth blocking merging over (though you have two approvals anyway)

[james-garner-canonical]

This sentence left me wondering what these different artefacts are specifically and what the implications are for someone doing a release.

Aren't all our docs always tracking main, rather than specific releases?

Are the release artifacts changing from just being zipped source code?

Oh also today I learned that artifacts is US English, and artefacts is British English.

[tonyandrewmeyer]

Suggested change

	`ops` and `ops-scenario` releases have several documentation artifacts, each
	serving a separate purpose and covering a different level.
	`ops` and `ops-scenario` releases have several documentation artefacts, each
	serving a separate purpose and covering a different level.

[tonyandrewmeyer]

This sentence left me wondering what these different artefacts are specifically and what the implications are for someone doing a release.

Hmm, I maybe need to use a different term then. They are the things outlined in the subsequent sections: raw logs, changelogs, release notes, release post.

Aren't all our docs always tracking main, rather than specific releases?

Yes, except for these documents that are specifically about the release.

Are the release artifacts changing from just being zipped source code?

The zipped and tar.gz repository dump (what GitHub calls "assets") stay the same: I don't think we can change those, and no-one should really be using them. The CI pipeline produces .tar.gz and wheels of the two packages (unchanged for ops, minor change for ops-scenario in that it previously was wheel-only) that are uploaded to PyPI and attestations for each of those (unchanged for ops, new for ops-scenario). We manually create the CHANGES.md files (unchanged for ops, new for ops-scenario), which are included in both the "assets" and PyPI artefacts. We also manually create the GitHub Releases text and Discourse post, which only live in those places.

Any suggestions for a better/clearer title here? Or clearer text?

[james-garner-canonical]

Oh I see now. I just immediately jumped to build artefacts. Probably just my unfamiliarity with terminology. Mentally substituting with "We produce several pieces of documentation for ops and ops-scenario releases, each serving ..." makes everything make sense for me.

[benhoyt]

I've had a look over this, and the changes themselves look okay to me, but I'm marking as "Request changes" as a "do not merge yet" signal, as I think we should have further discussion about whether this is the right approach as a team. In particular, the list of steps are getting quite unwieldy especially with the "if you're just releasing ops, do X" or "for ops-scenario, do Y" kind of stuff.

So let's discuss in The Hague and see if we can simplify. I'll add it to our list of discussion topics.

[tonyandrewmeyer]

So let's discuss in The Hague and see if we can simplify. I'll add it to our list of discussion topics.

I think the conclusion of that discussion was that we would look into further improvements and simplifications and automation.

In the meantime, I think we should merge this as the current state to work against, rather than what we have in main. It would also allow us to do the releasing in the meantime, even if it's not as ideal as we would perhaps like.