testing: release 30.20191014.1

[dustymabe]

First, verify that you meet all the prerequisites

Pre-release

Promote testing-devel changes

From the checkout for fedora-coreos-config (replace upstream below with
whichever remote name tracks coreos/):

• git fetch upstream
• git checkout testing
• git reset --hard upstream/testing
• /path/to/fedora-coreos-releng-automation/scripts/promote-config.sh testing-devel
• Sanity check promotion with git show
• Open PR against the testing branch on https://github.com/coreos/fedora-coreos-config
• Post a link to the PR as a comment to this issue
• Ideally have at least one other person check it and approve
• Once CI has passed, merge it

Build

• Start a pipeline build (select testing, and fill in version number using the N.YYYYMMDD.P format, pending finalization of Version numbering for OS releases fedora-coreos-tracker#81)
• Post a link to the job as a comment to this issue
• Wait for the job to finish

Sanity-check the build

Using the the build browser for the testing stream:

• Verify that the parent commit and version match the previous testing release (in the future, we'll want to integrate this check in the release job)
• Check kola AWS run to make sure it didn't fail

⚠️ Release ⚠️

IMPORTANT: this is the point of no return here. Once the OSTree commit is
imported into the unified repo, any machine that manually runs rpm-ostree upgrade will have the new update.

Importing OSTree commit

In the future, the OSTree commit import will be integrated in the release job.

• Open an issue on https://pagure.io/releng to ask for the OSTree commit to be imported (include a URL to the .sig which should be alongside the tarfile in the bucket and signed by the primary Fedora key)
• Post a link to the issue as a comment in this issue
• Wait for releng to process the request
• Verify that the OSTree commit and its signature are present and valid by booting a VM at the previous release (e.g. cosa run -d /path/to/previous.qcow2) and verifying that rpm-ostree upgrade works and rpm-ostree status shows a valid signature.

Run the release job

• Run the release job, filling in for parameters testing and the new version ID
• Post a link to the job as a comment to this issue
• Wait for job to finish

At this point, Cincinnati will see the new release on its next refresh and create a corresponding node in the graph without edges pointing to it yet.

Refresh metadata (stream and updates)

From a checkout of this repo:

• Update stream metadata, by running:

fedora-coreos-stream-generator -releases=https://fcos-builds.s3.amazonaws.com/prod/streams/testing/releases.json  -output-file=streams/testing.json -pretty-print

• Update the updates metadata, editing updates/testing.json:
  • Find the last-known-good release (whose rollout has a start_percentage of 100) and set its version to the most recent completed rollout
  • Delete releases with completed rollouts
  • Add a new rollout:
    • Set version field to the new version
    • Set start_epoch field to a future timestamp for the rollout start (e.g. date -d '2019/09/10 14:30UTC' +%s)
    • Set start_percentage field to 0.0
    • Set duration_minutes field to a reasonable rollout window (e.g. 2880 for 48h)
  • Update the last-modified field to current time (e.g. date -u +%Y-%m-%dT%H:%M:%SZ)

A reviewer can validate the start_epoch time by running date -u -d @<EPOCH>. An example of encoding and decoding in one step: date -d '2019/09/10 14:30UTC' +%s | xargs -I{} date -u -d @{}.

• Commit the changes and open a PR against the repo.
• Post a link to the PR as a comment to this issue
• Wait for the PR to be approved.
• Once approved, merge it and push the content to S3:

aws s3 sync --acl public-read --cache-control 'max-age=60' --exclude '*' --include 'streams/*' --include 'updates/*' . s3://fcos-builds

• Verify the new version shows up on the download page
• Verify the incoming edges are showing up in the update graph:

curl -H 'Accept: application/json' 'https://updates.coreos.stg.fedoraproject.org/v1/graph?basearch=x86_64&stream=testing&rollout_wariness=0'

NOTE: In the future, most of these steps will be automated and a syncer will push the updated metadata to S3.

[dustymabe]

PRs to testing branch:

• [testing] Add coreos-keep-cgroup-v1.service fedora-coreos-config#238
• [testing] Revert recent promotion to testing branch fedora-coreos-config#237

[dustymabe]

Build is running:

• https://jenkins-fedora-coreos.apps.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline/10164/

[dustymabe]

The "Build Aliyun" stage failed:

[fedora-coreos-fedora-coreos-pipeline] Running shell script

+ set -xeuo pipefail

+ coreos-assembler buildextend-aliyun

Unknown command: buildextend-aliyun

script returned exit code 1

I opened this to workaround hopefully: coreos/fedora-coreos-pipeline#169

[jlebon]

Without Aliyun: https://jenkins-fedora-coreos.apps.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline/10167

[jlebon]

The pipeline somehow picked the wrong cosa image. Rerunning in https://jenkins-fedora-coreos.apps.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline/10170/console.

[jlebon]

OK, we were hitting more issues with the wrong cosa image being pulled. Somehow fixed itself under threat of being debugged in https://jenkins-fedora-coreos.apps.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline/10173/console.

[dustymabe]

Outstanding request to releng: https://pagure.io/releng/issue/9062

[dustymabe]

release job running: https://jenkins-fedora-coreos.apps.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline-release/216/

[dustymabe]

and that failed with:

+ coreos-assembler aws-replicate --build=30.20191014.1

could not list regions: describing regions: UnauthorizedOperation: You are not authorized to perform this operation.

	status code: 403, request id: 4539c89f-5036-4932-8b69-8deb7c7844fa

We need to add more permissions for coreos/coreos-assembler@ab3cae5

[dustymabe]

Sent a patch to get added permissions: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/O5YA3HZ2NHJC67W75RRTEZT35GS2FUBV/

[jlebon]

Final release job that worked: https://jenkins-fedora-coreos.apps.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline-release/220

[jlebon]

#28

[jlebon]

This is done now. Keeping open to monitor rollout.

[lucab]

My canary rebooted into 30.20191014.1 at around Thu Nov 28 06:30:22 UTC 2019. The monitoring service on top of it (running in a podman-based systemd service) came back correctly.

[lucab]

Rollout successfully completed, closing.