Should Gateway API Standard Channel be Installed by Default by Kubernetes Distributions?

[robscott]

@shaneutt's earlier discussion about whether this project should move in-tree led to lots of great discussion. Although the idea was controversial, it hit on an important pain point for adoption of this API - it's not included by default in Kubernetes, and managing these CRDs on your own can become rather onerous.

In this discussion I'd like to explore another possible solution to that core problem: Requiring Kubernetes distributions to include Gateway API CRDs by default to be considered conformant. Instead of a tight coupling between Gateway API and Kubernetes versions, we'd aim for a slightly looser connection. Each Kubernetes release would have a corresponding requirement that a minimum version of Gateway API standard channel CRDs were included. Given our ongoing commitment of limiting standard channel to stable APIs with fully-backwards compatible changes, these upgrades would always be safe for Kubernetes distributions to support.

The rationale for this would be that:

1. Gateway API standard channel is already far more capable than Ingress, we should make it easier to use
2. Requiring a minimum version for each Kubernetes release leaves room for distributions or users to install newer versions of Gateway API CRDs
3. We still get to maintain our own independent release cycle and velocity
4. The latest version of Gateway API can still be installed on any Kubernetes cluster

If this proposal gains traction, there are a couple related problems we'd need to solve first:

1. We'd need to find a way to ensure that experimental APIs were handled separately so Kubernetes distributions were not installing over top of them. My preferred solution is covered in New Experimental APIs should start in x-k8s.io API Group #3497, but I'm open to alternatives.
2. We'd need to take this idea to the broader Kubernetes community to see if this idea was more broadly compelling.

[shaneutt]

In spirit I'm in favor. As Gateway API becomes more and more ubiquitous and as platforms want to do obvious things like provide their own default implementations while also supporting third party implementations, it is inevitable that those platforms are going to go ahead and implement life-cycle management for the CRDs (and something several of us in the community have been talking about for a while, cc @kflynn @JoelSpeed @mikemorris), and in fact we've already seen it.

In practice however, I don't think this is cut and dry. CRDs were not quite ready for the use case where something deployed via CRD is ubiquitous and used by multiple implementations. There are problems like "dead fields" and other things that need to be worked through.

Dead Fields: the situation where multiple implementations that need the same CRDs are technically "compatible" but one needs a newer version of the CRDs that contains fields the other does not support, and it "works" but those fields become "inert" when specified in a way that is not obvious or communicated to the user. In the worse case, this might be a security sensitive field like auth (which we have coming soon) which the user specifies and applies, and the implementation happily throws the auth away and deploys the rest quietly.

It's not clear to me that we're going to be able to get away with saying it's "required for platforms" unless we wanna go ahead and start talking in-tree again (which personally I think we're just dancing around that). In any case though what I do think we should be doing is:

• providing documented guidance for platforms on how we advise managing the CRDs
• enumerating the known problems with Gateway API CRDs in multi-implementation environments and providing upstream fixes that platforms can then employ
• splitting off everything experimental / not-GA off into its own API group

I think these are all things that will help create a more consistent user (and implementer) experience, and then the Kubernetes platforms of today (and tomorrow) will be better able to supply that. As far as making it required for platforms to supply that? Maybe in time?

[robscott]

unless we wanna go ahead and start talking in-tree again (which personally I think we're just dancing around that).

I'd disagree that my proposed approach has significant advantages over moving the APIs into k/k:

1. The latest version of the CRDs can be installed in any Kubernetes cluster
2. Gateway API maintains a separate independent release schedule
3. Experimental channel remains more accessible for most users (particularly using managed cluster providers) than alpha APIs in k/k

providing documented guidance for platforms on how we advise managing the CRDs
enumerating the known problems with Gateway API CRDs in multi-implementation environments and providing upstream fixes that platforms can then employ

+1, both of these are very necessary, and we can/should work on them in parallel to this longer term idea.

splitting off everything experimental / not-GA off into its own API group

+100, I've been trying to get some traction on this topic for quite awhile. I know it's a significant pain point for implementations, but I think it is a prerequisite for Gateway API being included on most Kubernetes clusters, and at least to me, the benefits involved seem to outweigh the work this would require.

[howardjohn]

Obviously its hard for anyone to say "no" to "should our software be forced to be installed everywhere by default" -- great for us -- more of a discussion for the people we are forcing it onto, though.

But there is one big risk here, and that is the flexibility. I think the intention of what you are saying is aligned with what i would want here, but I would want to make sure we codify it. Something like:

• The cluster must have a minimum version of X.
• The cluster must not prevent installing a custom versions

This has a lot of questions though. What happens if the cluster upgrades, and the user-installed CRD is tool old? Any mixed ownership stuff is wonky... what happens if the user deletes the CRD?

[kflynn]

I think I could probably be OK with the management plane ignoring anything in the experimental group.

I am absolutely not going to be OK with there being no way to install experimental. I may be OK with needing to spin up a new cluster to get experimental if you allow the cluster provider to automagically install standard when you create the cluster.

[mikemorris]

I am absolutely not going to be OK with there being no way to install experimental.

This is an outcome I'm very much interested in avoiding as well, to prevent similar lack of signal/feedback as in-tree Kubernetes alpha APIs due to limited availability.

[robscott]

I also want managed providers to be able to provide a safe way to install experimental, but if those managed providers are also including an implementation of Gateway API, it becomes very difficult to provide the same kind of support guarantees most of these platforms offer if the underlying API can change unexpectedly.

Frankly I think we're on a trajectory where more and more clusters are going to start coming with Gateway API installed by default, and I expect many of those cluster providers to choose option 1) above as option 2) is very difficult to provide any kind of support guarantees for.

IMO, separating experimental channel into a different API group helps experimental channel stay broadly accessible instead of becoming nearly as inaccessible as alpha APIs in k/k.

[howardjohn]

If we go with a hypothetical where we make the separate group - users can access it easier than alpha APIs, but that means nothing if implementations don't implement it. I cannot see myself signing up to double the number of APIs I am maintaining. If no one implements them, it doesn't matter how easy it is for the user to access them.

It seems like if users of a certain Kubernetes distribution are interested in usage of experimental that would provide motivation for vendors to not remove their ability to access them.

[mikemorris]

It seems like if users of a certain Kubernetes distribution are interested in usage of experimental that would provide motivation for vendors to not remove their ability to access them.

I'm sufficiently motivated to enable the Gateway API experimental channel if at all possible, but similarly constrained as @robscott due to policies and "supportability" requirements - we have plenty of users who similarly ask about Kubernetes alpha features, but we realistically can't consider offering those on any sort of mainstream cluster that could be used in production (I think GKE might offer some "disposable" clusters that can't be upgraded and shut down after 30 days?) and still be able to meet platform guarantees and user expectations.