decision: Evaluate options for primary container image storage

[youngnick]

Since Docker Hub has added pull rate limits, we need to reevaluate if we should be using it as our canonical container repository.

Options here:

• Stay with Docker Hub, hope nobody hits rate limits.
• Github Container Registry
• Google Container Registry (as used by Kubernetes)
• CNCF hosted Harbor (if there even is one)
• Something else

[stevesloka]

// ref #3184

[danehans]

@youngnick @stevesloka do you have a milestone that you would like to target for this?

[youngnick]

I don't think there's any huge rush, just needs a writeup about the tradeoffs of the various options. This feels like a "sometime in the next few releases" issue for me. Maybe we could say "by 1.16"?

[xaleeks]

Nick, let’s try to land this in v1.16? I recommend http://ghcr.io/

Interesting that you mentioned Kubernetes project, they are actually trying to use harbor and possibly other registries for a better experience where any pull from k8s.registry.io is routed to the nearest cloud provider PoP to pull from that registry.

See kubernetes/k8s.io#1834

[youngnick]

Yes, I think that GHCR sounds fine to me, and that 1.16 is a good timeframe.

[youngnick]

Slipping this one to 1.17.

[skriss]

GitHub Packages Container Registry is now GA: https://github.blog/2021-06-21-github-packages-container-registry-generally-available/

[youngnick]

As of this comment, we've decided that we'll move to Github Container Registry, so the next steps is to write up a design for switching to GHCR, which should include:

• How long do we continue pushing images to Docker Hub?
• How do we encourage people to migrate?
  and so on.

[skriss]

Here's my plan:

1. Start pushing to GHCR in parallel with Docker Hub, for both main and tag builds. Continue to reference the Docker Hub image in examples & CI for now. This will let us confirm everything is working with GHCR.
2. Publicize that we'll be migrating to GHCR and we'll stop pushing to Docker Hub after release v1.2x (maybe v1.21?)
3. Continue pushing to both registries, but switch example YAML & all other refs over to GHCR. Ideally we can do this for the 1.19 release if everything's in order, to start the migration.
4. After the release we decide on from Step 2 (e.g. v1.21), turn off the Docker Hub push. Existing images will be left there indefinitely for posterity, but no new images will be pushed.

We may also want to manually push the last ~year's worth of release-tagged images from Docker Hub to GHCR, just so they're all there.

[sunjayBhatia]

We may also want to manually push the last ~year's worth of release-tagged images from Docker Hub to GHCR, just so they're all there.

yeah thats a good idea, just in case

[xaleeks]

Thanks @skriss , that's a very tempered approach to migration

just one comment about the last part, Dockerhub has retention policies in place that deletes images that do not get any pulls over some specified time (6 months I believe), so we might want to start moving some of the older images over soon if we want to keep them

[skriss]

Closing this decision issue out since we've agreed on a plan, will use #3998 to track execution.