Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

stable: new release on 2024-03-11 (39.20240225.3.0) #862

Closed
43 tasks done
c4rt0 opened this issue Feb 27, 2024 · 6 comments
Closed
43 tasks done

stable: new release on 2024-03-11 (39.20240225.3.0) #862

c4rt0 opened this issue Feb 27, 2024 · 6 comments
Assignees
@marmijo
Labels
jira kind/release ok-to-promote

Comments

@c4rt0
Copy link
Member

c4rt0 commented Feb 27, 2024 •
edited by marmijo
Loading

First, verify that you meet all the prerequisites

Edit the issue title to include today's date. Once the pipeline spits out the new version ID, you can append it to the title e.g. (31.20191117.3.0).

Pre-release

Promote testing changes to stable

Manual alternative

Sometimes you need to run the process manually like if you need to add an extra commit to change something in manifest.yaml. The steps for this are:

Build

Sanity-check the build

Using the the build browser for the stable stream:

  • Verify that the parent commit and version match the previous stable release (in the future, we'll want to integrate this check in the release job)
    • x86_64
    • aarch64
    • ppc64le
    • s390x
  • Check kola extended upgrade runs to make sure they didn't fail
    • x86_64
    • aarch64
    • ppc64le
    • s390x
  • Check kola AWS runs to make sure they didn't fail
    • x86_64
    • aarch64
  • Check kola OpenStack runs to make sure they didn't fail
    • x86_64
    • aarch64
  • Check kola Azure run to make sure it didn't fail
    • x86_64
  • Check kola GCP runs to make sure they didn't fail
    • x86_64
    • aarch64

⚠️ Release ⚠️

IMPORTANT: this is the point of no return here. Once the OSTree commit is
imported into the unified repo, any machine that manually runs rpm-ostree upgrade will have the new update.

Run the release job

  • Run the release job, filling in for parameters stable and the new version ID
  • Post a link to the job as a comment to this issue
  • Wait for job to finish

At this point, Cincinnati will see the new release on its next refresh and create a corresponding node in the graph without edges pointing to it yet.

Refresh metadata (stream and updates)

  • Wait for all releases that will be released simultaneously to reach this step in the process
  • Go to the rollout workflow, click "Run workflow", and fill out the form
Rollout general guidelines
Risk Day of the week Rollout Start Time Time allocation
risky Tuesday 2PM UTC 72H
common Tuesday 2PM UTC 48H
rapid Tuesday 2PM UTC 24H

When setting a rollout start time ask "when would be the best time to react to
any errors or regressions from updates?". Commonly we select 2PM UTC so that the
rollout's start at 10am EST(±1 for daylight savings), but these can be fluid and
adjust after talking with the fedora-coreos IRC. Note, this is impacted by the
day of the week and holidays.

The later in the week the release gets held up due to unforeseen issues the more
likely the rollout time allocation will need to shrink or the release will need
to be deferred.

Manual alternative
  • Make sure your fedora-coreos-stream-generator binary is up-to-date.

From a checkout of this repo:

  • Update stream metadata, by running:
fedora-coreos-stream-generator -releases=https://fcos-builds.s3.amazonaws.com/prod/streams/stable/releases.json  -output-file=streams/stable.json -pretty-print
  • Add a rollout. For example, for a 48-hour rollout starting at 10 AM ET the same day, run:
./rollout.py add stable <version> "10 am ET today" 48
  • Commit the changes and open a PR against the repo
Update graph manual check 
curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=x86_64&stream=stable&rollout_wariness=0'
curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=aarch64&stream=stable&rollout_wariness=0'
curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=ppc64le&stream=stable&rollout_wariness=0'
curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=s390x&stream=stable&rollout_wariness=0'

NOTE: In the future, most of these steps will be automated.

Housekeeping

  • If one doesn't already exist, open an issue in this repo for the next release in this stream. Use the approximate date of the release in the title.
  • Issues opened via the previous link will automatically create a linked Jira card. Assign the GitHub issue and Jira card to the next person in the rotation.
@coreosbot-releng coreosbot-releng mentioned this issue Mar 11, 2024
Merged
@marmijo
Copy link
Member

marmijo commented Mar 11, 2024 •
edited
Loading

Build Jobs:

  • x86_64 ❌
    • kola test ext.config.boot.bootupd failed during the build.
  • x86_64 ❌
    • kola test ext.config.boot.bootupd failed during the build as well. See the below comment for the resolution

@jlebon
Copy link
Member

jlebon commented Mar 12, 2024

Due to coreos/bootupd#609 (comment), I pushed https://github.com/coreos/coreos-assembler/tree/fcos-39.20240225.3.0, which is the cosa from the previous testing release plus this commit.

Building in https://jenkins-fedora-coreos-pipeline.apps.ocp.fedoraproject.org/blue/organizations/jenkins/build-cosa/detail/build-cosa/755/pipeline. Once that's done, rebuild this release with COREOS_ASSEMBLER_IMAGE set to quay.io/coreos-assembler/coreos-assembler:fcos-39.20240225.3.0.

@marmijo
Copy link
Member

marmijo commented Mar 12, 2024

SGTM, thanks for your help @jlebon!

@marmijo
Copy link
Member

marmijo commented Mar 12, 2024 •
edited
Loading

New build job:

jlebon added a commit to jlebon/coreos-assembler that referenced this issue Mar 12, 2024
@jlebon
create_disk: run bootupctl backend install using bootupd from targe…
514fb10 
…t system

Just like most tools run in this script, we want to use the version of
bootupd in the target system rather than whatever happens to be in cosa
to ensure maximum compatibility.

This is already fixed in the osbuild flow. So this is just fixing it in
the legacy flow until we've fully moved over to osbuild since it's easy
to do. While we're here, drop the `unshare` workaround since it's no
longer needed.

This bit us recently:

coreos/fedora-coreos-streams#862 (comment)
@jlebon jlebon mentioned this issue Mar 12, 2024
Merged
@marmijo marmijo changed the title stable: new release on 2024-03-11 stable: new release on 2024-03-11 (39.20240225.3.0) Mar 12, 2024
jlebon added a commit to coreos/coreos-assembler that referenced this issue Mar 12, 2024
@jlebon
create_disk: run bootupctl backend install using bootupd from targe…
558bf5f 
…t system

Just like most tools run in this script, we want to use the version of
bootupd in the target system rather than whatever happens to be in cosa
to ensure maximum compatibility.

This is already fixed in the osbuild flow. So this is just fixing it in
the legacy flow until we've fully moved over to osbuild since it's easy
to do. While we're here, drop the `unshare` workaround since it's no
longer needed.

This bit us recently:

coreos/fedora-coreos-streams#862 (comment)
@marmijo
Copy link
Member

marmijo commented Mar 13, 2024

AWS Azure GCP OpenStack
x86_64 ✔️ ✔️ ✔️ ✔️
aarch64 ✔️ − ✔️ ✔️

@marmijo
Copy link
Member

marmijo commented Mar 13, 2024

Release job: https://jenkins-fedora-coreos-pipeline.apps.ocp.fedoraproject.org/blue/organizations/jenkins/release/detail/release/1185/pipeline

@marmijo marmijo closed this as completed Mar 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@marmijo marmijo

Labels
jira kind/release ok-to-promote
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jlebon @c4rt0 @marmijo