k8s.gcr.io VDF (Vanity Domain Flip): Move official container images to K8s Infra

[jbeda]

Google Infra --> K8s Infra tasks (ordered by priority):

• Ensure that k8s-artifacts-prod GCR is a superset of google-containers (aka backfiling; this has already been done and is easy to check before VDF happens)
• Fix auditor's ability to handle child images in fat manifests (issue)
• Optimize backup mechanism (issue)
• Add CLI command to make editing the promoter manifests easier (issue)
• Auditor alerts --> <somewhere>

Release Engineering

• (releng: Add new projects for staging/releasing Kubernetes k8s.io#624) Create k8s-staging-kubernetes GCP project (prod staging project)
• (backfill images from google-containers k8s.io#623) Backfill prod k8s.gcr.io images
• Figure out image promotion for staging
  • (releng: Use a temp namespace for k8s-staging-kubernetes promotion k8s.io#645) Configure k8s-staging-kubernetes promoter-manifest to point to test prod namespace
  • (Grant kubernetes-release-test GCB write on k8s-staging-kubernetes GCR k8s.io#673, Empower old release GCB to push to staging k8s.io#675) Grant kubernetes-release-test GCB write on k8s-staging-kubernetes GCR
  • ([VDF] anago: Publish container images on K8s Infra (k8s-staging-kubernetes) #1199) Publish container images on K8s Infra (k8s-staging-kubernetes)
  • Add CLI command to dump and merge an image manifest for promotion - [Feature request] Add CLI command to dump and merge an image manifest for promotion kubernetes-sigs/promo-tools#190

---

Original note from @jbeda:

We should not be using a Google only repo for official k8s builds.

[ixdy]

Yep, this is something that @timstclair and I are looking into.

[timstclair]

We're expecting to have a proposal out in the next week or two that will address this.

[zmerlynn]

Please include me. We have internal infrastructure that handles the replication of google-containers to non-US regions.

[luxas]

Please include me as well, I have been thinking about this for some time as well, but haven't just had the time to focus fully on trying to make a change here (not that I have the permissions to do so either but 😄)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[david-mcmahon]

cc @javier-b-perez

[luxas]

Progress: kubernetes/kubernetes#54174

/remove-lifecycle stale

cc @thockin @ixdy

What are the next steps here?
To start moving over images to a specific Kubernetes suborg gradually and/or updating refs in all other repos as well?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[stp-ip]

/remove-lifecycle stale
/lifecycle frozen

[listx]

I really hope this will happen in the coming months. Check out this issue for high-level updates, but also updates to the manifest.yaml files here, as this is the first batch of staging registries that will be using the new non-Googler promotion process.

Or, please attend the wg-k8s-infra meetings where updates to this work is currently on the agenda.

[justaugustus]

/assign @tpepper @justaugustus @timothysc @dims
/milestone v1.15
/priority critical-urgent
/area release-eng

[justaugustus]

@listx -- Would you mind providing an update on your side as we're nearing the date?

[listx]

I'm manually performing the backups for now to make the automated jobs not time-out and cause a quota exceeded error. The backup optimization implementation has hit a snag with Workload Identity here kubernetes/k8s.io#677, but that should be sorted out by EOD tomorrow.

The auditor has had fixes to child manifest detection and this has been rolled out earlier today.

There is an internal Google change that will start rolling out on Wednesday (April 1) morning for the flip itself. As k8s-artifacts-prod has been a superset of google-containers for a while now, the transition should be seamless. (However, I will perform another backfill if necessary tomorrow).

The backup stuff is in a bit of a pinch but I don't think it will take longer than tomorrow to sort out (in time for the flip on Wed morning).

[justaugustus]

@listx -- Thanks for the update! I think we're fine for day 1 as long as the backfill is set (which it is) and we've got a go-forward plan for the backups (which we do).

On my end, I've successfully tested the Releng changes here. I'm going to do a few spot-checks just in case, but we should be all good to go! :)

[listx]

The flip is in progress. Charts and updates haven been posted here: https://groups.google.com/d/msg/kubernetes-sig-release/ew-k9PEBckQ/mSa7KGeUCAAJ

[listx]

Forgot to update this thread, but TL;DR: the flip has been rolled back due to an unforeseen issue. Details here: https://groups.google.com/d/msg/kubernetes-sig-release/ew-k9PEBckQ/SZC2KeFYAwAJ

We will attempt the flip again in the coming weeks; I will update this thread again once we decide on the date.

[justaugustus]

Last I chatted w/ @listx ETA is in a few weeks.
They're still working on some stuff on the Google side.

[thockin]

I gave a quick update at the WG meeting. TL;DR is that over time, bad assumptions grow deep roots. We're disentangling the roots now, but it reaches across several teams, and we've already caused one problem with this, so "once bitten, twice shy" is in full effect. We believe we're close. O(weeks) seems right.

…

On Wed, May 13, 2020 at 8:12 PM Stephen Augustus ***@***.***> wrote: Last I chatted w/ @listx <https://github.com/listx> ETA is in a few weeks. They're still working on some stuff on the Google side. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#270 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVDEWRZ4DKMLMOMEW53RRNORLANCNFSM4DCOKK6A> .

[listx]

I vote to close this as VDF was completed on July 24, 2020.