Deprecate and migrate away from gs://kubernetes-release

[spiffxp]

Part of umbrella issue to migrate the kubernetes project away from use of GCP project google-containers: #1571

This issue covers the deprecation of and migration away from the following google.com assets:

• the google.com-owned GCS bucket gs://kubernetes-release living in GCP project google-containers, in favor of the community-owned GCS bucket gs://k8s-release living in GCP project TBD (currently k8s-release)
• the region-specific GCS buckets gs://kubernetes-release-asia and gs://kubernetes-release-eu, same as above but gs://k8s-release-eu and gs://k8s-release-asia instead
• TODO: are there container images involved here as well, or did we already address that with k8s.gcr.io?

These are not labeled as steps just yet because not everything needs to be completed to full fidelity in strict sequential order. I would prefer that we get a sense sooner rather than later what the impact of shifting dl.k8s.io traffic will be, in terms of how much budget, and what percentage of traffic that represents vs. hardcoded traffic.

Determine new-to-deprecated sync implementation and deprecation window

There are likely a lot of people out there that have gs://kubernetes-release hardcoded. It's unreasonable to stop putting new releases there without some kind of advance warning. So after announcing our intent to deprecate gs://kubernetes-release, we should decide how we're going to sync new releases back there (and its region-specific buckets)

• gsutil rsync
• Google Cloud Storage Transfer Service
• etc.

As for the deprecation window itself, I think it's fair to treat this with a deprecation clock equivalent to disabling a v1 API.

Determine gs://k8s-release project location and geo-sync implementation

• Someone (probably me) manually created gs://k8s-release and its other buckets to prevent someone else from grabbing the name
• The -eu and -asia buckets are not actually region-specific, and should be recreated as such
• We should decide how we're going to implement region syncing (same as above)
• We should decide at this stage whether we want to block on a binary artifact promotion process, or get by with one of the syncing mechanisms from above

Use dl.k8s.io where possible and identify remaining hardcoded bucket name references across the project

The only time a kubernetes release artifact GCS bucket name needs to show up in a URI is if gsutil is involved, or someone is explicitly interested in browsing the bucket. For tools like curl or wget that retrieve binaries via HTTP, we have https://dl.k8s.io, which will allow us to automatically shift traffic from one bucket to the next depending on the requested URIs

I started doing this for a few projects while working on #2318, e.g.

• use dl.k8s.io where possible cloud-provider-gcp#252
• 🌱 Use dl.k8s.io instead of hardcoded GCS URIs kubernetes-sigs/cluster-api#4958

TODO: a cs.k8s.io query and resulting checklist of repos to investigate

Shift dl.k8s.io traffic to gs://k8s-release-dev

TODO: there is a separate issue for this.

We will pre-seed gs://k8s-release with everything in gs://kubernetes-release, and gradually modify dl.k8s.io to redirect more and more traffic to gs://k8s-release.

The idea is not to flip a switch, just in case that sends us way more traffic than our budget is prepared to handle. Instead, let's consider shifting traffic gradually for certain URI patterns, or a certain percentage of requests, etc. It's unclear whether this will be as straightforward as adding lines to nginx, or whether we'll want GCLB changes as well.

Change remaining project references to gs://k8s-release

/area artifacts
/area prow
/area release-eng
/sig release
/sig testing
/wg k8s-infra
/priority important-soon
/kind cleanup
/milestone v1.23

[puerco]

/cc @kubernetes/release-engineering

[spiffxp]

Blocked on #1375

[spiffxp]

/milestone v1.24

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ameukam]

/remove-lifecycle stale

[ameukam]

/milestone clear
/lifecycle frozen
/priority backlog

[BenTheElder]

This is ~done. After the next round of patch releases, we will drop write permissions to this old bucket.

The new bucket is private and shielded behind the CDN donated to us by Fastly, which we are using to host binary downloads going forward.

Thanks @ameukam for pushing this forward with the release engineering team and branch managers.

[rittneje]

@BenTheElder Why was this change not mentioned anywhere? We only found out when we tried to download kubectl v1.29.9 and it didn't work. This should at least have been included in the changelog.

[sftim]

We did publish https://kubernetes.io/blog/2023/08/15/pkgs-k8s-io-introduction/ @rittneje

If you used anything other than the official source: you should have switched. We don't formally changelog infrastructure changes in the release notes, but we deliberately timed the pkgs.k8s.io article to come out on the same day as https://kubernetes.io/blog/2023/08/15/kubernetes-v1-28-release/

[sftim]

We also published https://kubernetes.io/blog/2023/06/09/dl-adopt-cdn/ earlier in 2023

[BenTheElder]

@BenTheElder Why was this change not mentioned anywhere? We only found out when we tried to download kubectl v1.29.9 and it didn't work. This should at least have been included in the changelog.

Kubernetes has been advertising all releases on dl.k8s.io for a LONG time now. We did not advertise them as being on gs://kubernetes-release.

gs://kubernetes-release used to be an implementation detail of dl.k8s.io, and we announced that we'd be serving through fastly more than a year ago. Now we're no longer publishing to it and only to the new community owned CDN backing bucket which is not public read, please use the CDN. kubernetes-release is an old GCS bucket internal to google's GCP projects so depending on it prevents us from putting the community at large in control of releases (which google is very much still funding, but is not solely controlling).

You should not depend on any non-documented details of Kubernetes or the project's infrastructure.

See also:
kubernetes/kubernetes#127796 (comment)
https://kubernetes.io/blog/2023/06/09/dl-adopt-cdn/ (which had a banner on the kubernetes.io website)
https://registry.k8s.io#stability

[BenTheElder]

We still need to phase out the related GCB bucket and service account / project. (Which are further implemntation details of releasing). Thread in https://kubernetes.slack.com/archives/CJH2GBF7Y/p1729677695289489

[ameukam]

@BenTheElder We can drop the IAM bindings for this bucket and leave it in read-only.

[BenTheElder]

IAM bindings are being dropped, this is done.

[BenTheElder]

see also #1571 (comment)