Automate binary promotion

[justinsb]

Opening per discussion with @ameukam in kOps office hours today.

We have jobs that automatically promote container images (that I believe are defined here. The kOps project has been running the binary promoter for about a year+ now, including proposing artifacts etc, but we have run it by hand.

The binary promoter was originally developed alongside the container-image-promoter, but has since moved into the release project (cc @justaugustus )

Now that we have more projects wanting to do binary promotion (e.g. the CSI project), and given that kOps has been using it successfully, we should complete the automation here by running the promotion itself automatically.

I think we "just" need to create appropriate jobs that mirror the CIP jobs but run the binary promoter instead.

[ameukam]

/area artifacts
/sig release
/area release-eng
/milestone v1.23

/assign

[justaugustus]

@justinsb @ameukam -- Let's see if we can find some time to catch up next week on this.

I'll start poking on the RelEng side:
/assign

[justaugustus]

I'll start poking on the RelEng side:
/assign

Starting here: kubernetes/test-infra#23416

[justaugustus]

Status:

• kpromo image build pre- and postsubmit: releng: Add image building jobs for kpromo test-infra#23416
• Image building init for kpromo: kpromo: Initial image building configuration release#2231
• Test file promotion periodic job: releng: Add periodic job for file promotion test-infra#23417
• Break dependencies on packages in k/release:
  • Init object and gcli packages from k/release: Init object and gcli packages from k/release kubernetes-sigs/release-sdk#2
  • Migrate pkg/object and pkg/gcp/gcp.go to sigs.k8s.io/release-sdk: Migrate pkg/object and pkg/gcp/gcp.go to sigs.k8s.io/release-sdk release#2232
• Migrate kpromo/cip-mm tools/packages into repo: Migrate kpromo/cip-mm tools/packages into repo kubernetes-sigs/promo-tools#407
• Tidy image building configs
  • kpromo: Initial merge of image building files kubernetes-sigs/promo-tools#408
  • releng: Tidy kpromo job configs test-infra#23430

Slack thread: https://kubernetes.slack.com/archives/CJH2GBF7Y/p1630416874004700

[justaugustus]

Status comment is here, but just to say a little more...

I've restored all of the promotion tools to the CIP repo, which is easier to do now that bazel has been removed (thanks @tylerferrara + @listx).

kpromo images build in presubmit (test env) and postsubmit (staging env).

There's a test periodic job for file promotion, which is analogous to the image promoter periodic (but uses debug values).

I tried a few things locally just to check out the outputs...

Release file(s) (do exist in prod)

❯ time docker run -v $(pwd):/workspace -it gcr.io/k8s-staging-artifact-promoter/kpromo-amd64:v0.1.0-1 run files --filestores=/workspace/filestores/k8s-staging-kops/filepromoter-manifest.yaml --files=/workspace/manifests/k8s-staging-kops/1.21.1.yaml --dry-run=true
********** START (DRY RUN) **********
INFO processing destination "gs://k8s-artifacts-prod/binaries/kops/"
INFO listing files in bucket k8s-staging-kops with prefix "kops/releases/"
INFO listing files in bucket k8s-artifacts-prod with prefix "binaries/kops/"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/darwin/amd64/kops"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/darwin/amd64/kops.sha256"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/images/dns-controller-amd64.tar.gz"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/images/dns-controller-amd64.tar.gz.sha256"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/images/dns-controller-arm64.tar.gz"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/images/dns-controller-arm64.tar.gz.sha256"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/images/images.tar.gz"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/images/images.tar.gz.sha256"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/images/kops-controller-amd64.tar.gz"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/images/kops-controller-amd64.tar.gz.sha256"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/images/kops-controller-arm64.tar.gz"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/images/kops-controller-arm64.tar.gz.sha256"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/images/kube-apiserver-healthcheck-amd64.tar.gz"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/images/kube-apiserver-healthcheck-amd64.tar.gz.sha256"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/images/kube-apiserver-healthcheck-arm64.tar.gz"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/images/kube-apiserver-healthcheck-arm64.tar.gz.sha256"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/linux/amd64/channels"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/linux/amd64/channels.sha256"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/linux/amd64/kops"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/linux/amd64/kops.sha256"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/linux/amd64/nodeup"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/linux/amd64/nodeup.sha256"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/linux/amd64/protokube"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/linux/amd64/protokube.sha256"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/linux/arm64/channels"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/linux/arm64/channels.sha256"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/linux/arm64/kops"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/linux/arm64/kops.sha256"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/linux/arm64/nodeup"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/linux/arm64/nodeup.sha256"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/linux/arm64/protokube"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/linux/arm64/protokube.sha256"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/windows/amd64/kops.exe"
INFO metadata match for "gs://k8s-artifacts-prod/binaries/kops/1.21.1/windows/amd64/kops.exe.sha256"
********** FINISHED (DRY RUN) **********
docker run -v $(pwd):/workspace -it  run files   --dry-run=true  0.13s user 0.11s system 8% cpu 2.627 total

Release file(s) (do NOT exist in prod)

❯ time docker run -v $(pwd):/workspace -it gcr.io/k8s-staging-artifact-promoter/kpromo-amd64:v0.1.0-1 run files --filestores=/workspace/filestores/k8s-staging-kops/filepromoter-manifest.yaml --files=/workspace/manifests/k8s-staging-kops/1.23.0-alpha.1.yaml --dry-run=true
********** START (DRY RUN) **********
INFO processing destination "gs://k8s-artifacts-prod/binaries/kops/"
INFO listing files in bucket k8s-staging-kops with prefix "kops/releases/"
INFO listing files in bucket k8s-artifacts-prod with prefix "binaries/kops/"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/darwin/amd64/kops" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/darwin/amd64/kops"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/darwin/amd64/kops.sha256" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/darwin/amd64/kops.sha256"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/darwin/arm64/kops" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/darwin/arm64/kops"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/darwin/arm64/kops.sha256" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/darwin/arm64/kops.sha256"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/images/dns-controller-amd64.tar.gz" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/images/dns-controller-amd64.tar.gz"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/images/dns-controller-amd64.tar.gz.sha256" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/images/dns-controller-amd64.tar.gz.sha256"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/images/dns-controller-arm64.tar.gz" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/images/dns-controller-arm64.tar.gz"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/images/dns-controller-arm64.tar.gz.sha256" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/images/dns-controller-arm64.tar.gz.sha256"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/images/images.tar.gz" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/images/images.tar.gz"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/images/images.tar.gz.sha256" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/images/images.tar.gz.sha256"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/images/kops-controller-amd64.tar.gz" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/images/kops-controller-amd64.tar.gz"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/images/kops-controller-amd64.tar.gz.sha256" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/images/kops-controller-amd64.tar.gz.sha256"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/images/kops-controller-arm64.tar.gz" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/images/kops-controller-arm64.tar.gz"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/images/kops-controller-arm64.tar.gz.sha256" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/images/kops-controller-arm64.tar.gz.sha256"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/images/kube-apiserver-healthcheck-amd64.tar.gz" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/images/kube-apiserver-healthcheck-amd64.tar.gz"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/images/kube-apiserver-healthcheck-amd64.tar.gz.sha256" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/images/kube-apiserver-healthcheck-amd64.tar.gz.sha256"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/images/kube-apiserver-healthcheck-arm64.tar.gz" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/images/kube-apiserver-healthcheck-arm64.tar.gz"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/images/kube-apiserver-healthcheck-arm64.tar.gz.sha256" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/images/kube-apiserver-healthcheck-arm64.tar.gz.sha256"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/linux/amd64/channels" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/linux/amd64/channels"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/linux/amd64/channels.sha256" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/linux/amd64/channels.sha256"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/linux/amd64/kops" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/linux/amd64/kops"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/linux/amd64/kops.sha256" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/linux/amd64/kops.sha256"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/linux/amd64/nodeup" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/linux/amd64/nodeup"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/linux/amd64/nodeup.sha256" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/linux/amd64/nodeup.sha256"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/linux/amd64/protokube" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/linux/amd64/protokube"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/linux/amd64/protokube.sha256" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/linux/amd64/protokube.sha256"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/linux/arm64/channels" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/linux/arm64/channels"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/linux/arm64/channels.sha256" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/linux/arm64/channels.sha256"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/linux/arm64/kops" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/linux/arm64/kops"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/linux/arm64/kops.sha256" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/linux/arm64/kops.sha256"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/linux/arm64/nodeup" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/linux/arm64/nodeup"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/linux/arm64/nodeup.sha256" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/linux/arm64/nodeup.sha256"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/linux/arm64/protokube" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/linux/arm64/protokube"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/linux/arm64/protokube.sha256" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/linux/arm64/protokube.sha256"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/windows/amd64/kops.exe" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/windows/amd64/kops.exe"
COPY "gs://k8s-staging-kops/kops/releases/1.23.0-alpha.1/windows/amd64/kops.exe.sha256" to "gs://k8s-artifacts-prod/binaries/kops/1.23.0-alpha.1/windows/amd64/kops.exe.sha256"
********** FINISHED (DRY RUN) **********
docker run -v $(pwd):/workspace -it  run files   --dry-run=true  0.12s user 0.15s system 9% cpu 2.820 total

Full project directory

❯ time docker run -v $(pwd):/workspace -it gcr.io/k8s-staging-artifact-promoter/kpromo-amd64:v0.1.0-1 run files --filestores=/workspace/filestores/k8s-staging-kops/filepromoter-manifest.yaml --files=/workspace/manifests/k8s-staging-kops --dry-run=true
********** START (DRY RUN) **********
INFO processing destination "gs://k8s-artifacts-prod/binaries/kops/"
INFO listing files in bucket k8s-staging-kops with prefix "kops/releases/"
INFO listing files in bucket k8s-artifacts-prod with prefix "binaries/kops/"
FATA run `kpromo run files`: error building operations: error building promotion operations for "gs://k8s-artifacts-prod/binaries/kops/": file "1.16.0/darwin/amd64/kops" not found in source ("gs://k8s-staging-kops/kops/releases/1.16.0/darwin/amd64/kops")
docker run -v $(pwd):/workspace -it  run files   --dry-run=true  0.14s user 0.13s system 9% cpu 2.830 total

tl;dr -- I see a few things to work on/ask questions about:

• Ensuring restrictions on top-level prod writes
• Support for multiple manifests (all staging projects) and multiple releases (per project)
• Enable manifests with artifacts that no longer exist on staging buckets to be skipped (we expect the staging artifacts to age out)
  • We should still report the issue, and:
  • Probably still call this a failure if any of the assets in a single manifest are not found i.e., if there were supposed to be 10 files in a 1.16 release but we only find 1, the copy should be skipped and reported on
• Should we allow backfill operations on a directory i.e., if the staging directory initially had 10 objects, but now has 15, do we care that 5 objects will be promoted on the next run?
  • Should this be configurable per project?

I'll be working on some of this in kubernetes-sigs/promo-tools#409.

[spiffxp]

/priority important-soon
/milestone v1.23

[justaugustus]

So we're capturing the repo-specific particulars, I've opened kubernetes-sigs/promo-tools#413.

I was blocked on some account auth issues, but after chatting w/ @justinsb, I've validated some next steps.
Stay tuned!

[justaugustus]

File promotion is live as of this weekend via kubernetes/test-infra#23571, https://github.com/kubernetes-sigs/k8s-container-image-promoter/releases/tag/v1.339.0, https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-k8sio-file-promo/1437149962537275392.

Docs are here: #2711

Please add any follow-ups to kubernetes-sigs/promo-tools#413.
/close

[k8s-ci-robot]

@justaugustus: Closing this issue.

In response to this:

File promotion is live as of this weekend via kubernetes/test-infra#23571.
Docs are here: #2711

Please add any follow-ups to kubernetes-sigs/promo-tools#413.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.