Skip to content

Latest commit

 

History

History
329 lines (213 loc) · 12.6 KB

index.md

File metadata and controls

329 lines (213 loc) · 12.6 KB

Creating a stable release

This document outlines the process for publishing a Charmed Kubernetes stable release.

Background

Repository layout

All charm repositories used by Charmed Kubernetes have a common branch scheme to provide a consistent experience across all code bases. Any external or shared repositories are forked into the charmed-kubernetes github organization and have the following branches:

  • main: The primary development branch. Merges are made against this branch as they are approved.
  • release_1.xx: The release branch. Major releases have main directly merged to release_1.xx. Bugfix releases have specific commits cherry-picked to release_1.xx.

Tags are used to mark releases on the release_1.xx branch.

Feature Freeze

2 weeks prior to a stable release the team goes into a feature freeze. At this time only bugfixes and concentration on resolving any other outstanding issues will take place for the first week of this freeze.

The remaining tasks will still be completed at the time of feature freeze giving Solutions QA a solid base to test from.

Conflict resolution

At the time of the feature freeze, new release_1.xx branches are created to match our default repo branches per the documentation below. During the feature freeze and Solutions QA period, fixes which need to be applied to address CI or QA failures (and only those specific fixes) are cherry-picked to the respective release branches.

Prepare CI

$stable++ release

It may feel early, but part of releasing the next stable version requires preparing for the release that will follow. This requires opening tracks and building relevant snaps and charms that will be used in the new edge channel.

For example, we requested 1.31 snap tracks while preparing for the 1.30 release:

Bundle/charm track requests are made by posting to the charmhub requests forum asking for new tracks to be opened for every necessary charm and bundle owned by Canonical Kubernetes. For example:

$stable release

Once upstream has an RC for the upcoming release, our CI should stop building pre-prelease snaps. This ensures the 1.xx track will end up with 1.xx.0 instead of 1.xx.1-alpha.0. For example, we merged the following between 1.29 RC and GA:

Additionally, if not done already, CI should include 1.xx in the version matrix and config for relevant jobs. For example, see these updates where we adjusted tests for our 1.29 release:

Preparing the release

Create release branches for all repos

Job: https://jenkins.canonical.com/k8s-ps5/job/cut-stable-release/

We need to create release_1.xx branches from main for all Charmed Kubernetes repositories. This will be our snapshot from which we test, fix, and subsequently promote to the new release.

Pin snap channel for charms in the release branches

We need to make sure that the kubernetes-<control-plane|e2e|worker> charms have 1.xx/stable set as the default snap channel. This should be done on each of the relevant git release_1.xx branches. For example, for the 1.29 GA:

Note: Changes to the above repos are required as some of our customers do not use our bundles for deployment.

Pin snap channel for bundles in the release branches

We need to make sure that the bundle fragments have 1.xx/stable set as the default snap channel on the release_1.xx branch. For example, for the 1.29 GA:

Note: Dont miss our badges like we've done so many times before!

Build charms and bundles from the release branches

Job: https://jenkins.canonical.com/k8s-ps5/job/build-charms/

This job clones the release_1.xx branch for each of our repos. It then builds each charm/bundle using those local repos. After building, they will be published to the 1.xx/beta channel in Charmhub based on the job options.

Note: This job must be run again if a subsequent commit is made to the release_1.xx branch of any component needed by this release.

Build 1.xx/beta charms and bundles

charm build options

Promote charms to latest/beta

Job: https://jenkins.canonical.com/k8s-ps5/job/promote-charms/

In preparation for running the validate-charm-release-upgrade job, the charms must be promoted from 1.xx/beta to latest/beta channels.

Note: This job must be run again if any charm needed by this release is rebuilt in the 1.xx/beta channel.

Promote 1.xx/beta charms to latest/beta

charm promote options

Promote new K8s snaps

Job: https://jenkins.canonical.com/k8s-ps5/job/build-snap-from-branch/

K8s snap promotion is handled by the sync-snaps job and will happen automatically after following the Prepare CI section above. If for some reason you need to manually build K8s snaps from a specific branch, use the above job with a branch parameter like 1.29.0.

The branch parameter gets translated to v$branch by snap.py which must correspond to a valid tag in our internal k8s mirror.

Internal verification

Run validate-charm-release-upgrade job

Job: https://jenkins.canonical.com/k8s-ps5/job/validate-charm-release-upgrade/

This validates the deployment using charms from the latest/stable channel, then performing an upgrade to latest/beta. The tests are parameterized to run on multiple series and with multiple snap channels.

Before running this job, confirm that the snap_version job parameter is set to the appropriate channel for this release (e.g. 1.29/beta).

Notify Solutions QA

At the end of the first week and assuming all major blockers are resolved, the release is passed over to Solutions QA (SQA) for sign-off. This is done by publishing a CI release with a new 1.xx tag and informing SQA of that tag. They will then have the remaining week to test and file bugs so engineering can work towards getting them resolved prior to GA.

Please note the Conflict Resolution Section for making any changes as a result of SQA testing.

Azure Arc Conformance

We certify Charmed Kubernetes with the Azure Arc program by running the following job on each new release:

https://jenkins.canonical.com/k8s-ps5/view/Conformance/job/conformance-arc-ck/

This job runs weekly, but publishing the results to Microsoft's bucket is not done by default. Ensure we have passing results, and run the above job with the "UPLOAD_RESULTS" parameter checked.

Credentials for this job are exported by the azure-arc.sh script, defined in the "Juju Data - Azure ARC" BitWarden entry, and delivered to all Jenkins workers as part of the juju_creds credential.

Note: The AZ_STORAGE_ACCOUNT_SAS key expires monthly and will need to be rotated via the partner credential portal; use the ARC_CANONICAL_GUID from the BitWarden entry to get a new value.

CNCF Conformance

Job: https://jenkins.canonical.com/k8s-ps5/job/conformance-cncf-ck/

Sync charmed-kubernetes/k8s-conformance main from upstream

Confirm passing results, then create a PR against the upstream k8s-conformance repo. For example, we used the following branch for CK 1.29:

And opened this upstream PR:

Note: CNCF requires a sign-off. After confirming results, issue a git commit --amend --signoff on the branch prior to submitting the PR.

Performing the release

Document release notes

  • Bugfixes
  • Enhancements
  • Known Limitations/Issues

Promote charms to stable

Job: https://jenkins.canonical.com/k8s-ps5/job/promote-charms/

This job takes a tag list, from_channel, and to_channel. A tag value of k8s would only promote the charms that make up core charmed-kubernetes. Ensure that k8s-operator is added to the tag list to include kubernetes operator charms. You typically run this job multiple times to get charms into all of the appropriate channels, for example:

  • 1.27/beta -> 1.27/stable
  • 1.27/beta -> latest/stable

Promote 1.xx/beta charms to stable channels

promote charm options

Build bundles to beta and stable

Job: https://jenkins.canonical.com/k8s-ps5/job/build-charms/

Bundles cannot be promoted because they reference specific channels at build time. Therefore, it's required to build bundles which reference the track charms and 1.xx/stable track snaps

Note: Run job two times, setting TO_CHANNEL as beta and stable

Note: The bundle filter shown below ensures only bundles are built when this job runs.

Build bundle Options

build bundle options

Extract the release bundle to our bundle repo

For reference purposes, we extract all Charmed Kubernetes bundles to our bundle repo. Use the release.sh script to extract the newly built bundle to the ./releases/$track directory and raise a PR. For example, for the 1.29 GA:

Confirm snap promotion from 1.xx/<risk> to latest/<risk>

Job: https://jenkins.canonical.com/k8s-ps5/job/sync-snaps/

This job will automatically promote snaps to latest. The only prereqs are that charms have been promoted and that the K8S_STABLE_VERSION enum is set to this release 1.xx. For example, for the 1.29 GA:

Note: Nightly charm and bundle builds will publish to both latest/edge and K8S_STABLE_VERSION/edge channels.

Tag release branches with the current stable bundle

Job: https://jenkins.canonical.com/k8s-ps5/job/sync-stable-tag-bundle-rev/

For all charm repos that make up CK, tag the existing release branches with the most recently released stable charmed-kubernetes bundle revision. Use the 1.xx/stable version number from charmhub.io/charmed-kubernetes, not the latest/stable version number.

Sync Stable Tag Bundle Rev Options

sync stable tag bundle rev options

Send announcement

Email announcement to k8s-crew with any relevant information.

Post Release

When $stable++ tracks are open, add them to our CI enumerations as well as our custom snap jobs. For example, see the additions made during our 1.29 GA to support the future 1.28 release:

Set cdk-addons envars

Update cdk-addons release-1.xx Makefile, e.g.:

Update cdk-addons main Makefile

Bump snap channel to the $stable++ release

Bump the kubernetes-<control-plane|e2e|worker> charms and bundle fragments in the main git branches to the future $stable++ release, e.g. 1.30/edge. You don't have to do this right away; in fact, you should wait until you actually have snaps in the $stable++/edge channels before making this change.

Adjust LP milestones

Run the [close|open]-milestone.py scripts found in the cdk-scripts repo repository. For example:

./close-milestone.py 1.29
./create-milestone.py 1.29+ck1

Fin