Replies: 8 comments 23 replies
-
|
One thing to clarify here: This is just for new resources, not for new fields on graduated resources, right? |
Beta Was this translation helpful? Give feedback.
-
|
Yep, just new experimental resources. When these new resources graduate to GA, we transition from x-k8s.io/v1alpha1 in experimental channel to k8s.io/v1 in both channels. |
Beta Was this translation helpful? Give feedback.
-
|
Why not both ? And why move ? |
Beta Was this translation helpful? Give feedback.
-
I think ^ is honestly the more difficult problem to solve tbh, I was kinda starting to come around to the proposal back in #2912 if we could drive sufficient improvements to the developer experience for watching two different groups for a similar resource. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, I agree that new fields is way harder to solve, but if new resources in a separate API works out well enough, we could later look at prototyping fields in v1 copies of objects in the experimental API group. If the separate group developer and user experience turns out okay. |
Beta Was this translation helpful? Give feedback.
-
|
I would love to do both, but ultimately there was sufficient pushback to that idea that this seemed like it may be a good compromise. This helps with some of the most critical issues:
I'm hoping this experiment goes well enough that there's interest in taking it further, but this is a fairly low risk step in what I think is the right direction. |
Beta Was this translation helpful? Give feedback.
-
|
@robscott would be good to also gather feedback from implementations on who will support this new |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, thanks for the clarifications here and on chat. I am good with this provided we do as much as possible to make experimental truly unstable, notably the expectation that implementations should not support experimental and v1 concurrently (I am not sure if we can somehow enforce this, since its kind of a prisoners dilemma -- see companies selling LTS, once one does it (for $$$ usually) there is an expectation others do it too). I do, however, have concerns about the proposals that use this as a stepping stone, around "Kubernetes comes with the CRDs enabled by default and you cannot get experimental fields". That certainly doesn't need to be linked to this decision though. |
Beta Was this translation helpful? Give feedback.
-
Do you mean concurrently in the sense of "actively for a single installation within a cluster" (this feels like it might be possible to enforce with conformance tests, and could possibly encourage safer patterns like using dev/staging clusters for testing experimental functionality), or "within a project's codebase"? |
Beta Was this translation helpful? Give feedback.
-
|
@mikemorris the big concern with splitting groups (from a maintainer POV) is the cost to support 2 groups is orders of magnitude larger than to support 2 versions of the same group (since versions are identical). So I really don't want to get into a world where an implementation has to deal with that. The easy way out is... say GW v1.5 promotes FooRoute. So v1.4 had xFooRoute. v1.5 should remove xFooRoute and add FooRoute (stable). Implementations supporting v1.5 should replace xFooRoute support with FooRoute support. tl;dr "within a projects codebase" |
Beta Was this translation helpful? Give feedback.
-
I remember this coming up before and I think there was some consideration that it might be possible to work with SIG API Machinery or improve tooling to make this easier?
That wouldn't solve for new experimental fields on existing stable resources though I think? |
Beta Was this translation helpful? Give feedback.
-
Correct -- this proposal is not covering that though |
Beta Was this translation helpful? Give feedback.
-
|
We discussed this at Kubecon, but I forgot to make a post here. I'm generally in favor of this proposal. In a future where platforms are increasingly deploying Gateway API CRDs by default, this provides flexibility to users and implementations who want to try out experimental features on those platforms. |
Beta Was this translation helpful? Give feedback.
-
|
Posted in slack but I'll surface this here Circling back to the discussion on which group |
Beta Was this translation helpful? Give feedback.
-
|
Yeah this specific proposal doesn't really provide isolation for experimental fields. My previous proposal that did (#3106) was not particularly popular, so I'm trying to start smaller here. |
Beta Was this translation helpful? Give feedback.
-
|
Agreed this is a bit awkward for features spanning across new resources and additions to existing resources, where perhaps either resources from the x-k8s.io group would be available but only alongside standard channel resources, or new fields on existing experimental channel resources in the newest release of the main group may become available without support in the implementation for x-k8s.io resources. This could lead to "dead fields" as @shaneutt describes in #3576 (comment) |
Beta Was this translation helpful? Give feedback.
-
|
To clarify, the changes I'm proposing here are quite tiny. The only difference from today is that new resources in experimental channel would start in a different experimental API group. Everything else about experimental channel would stay the same. In practice that means that installing experimental channel would continue to get you resources were fully experimental, and resources that were GA with experimental fields. Although I'd like to pursue broader changes like a full separation between release channels to eliminate problems like dead fields, this specific proposal is really small. |
Beta Was this translation helpful? Give feedback.
-
|
Gotcha, that makes sense - I was jumping ahead a bit to thinking about which resources a cloud provider might allow or install by default on a cluster. |
Beta Was this translation helpful? Give feedback.
-
|
The APIs represent a common way to solve a specific problem. If 2
implementations use the same API - and 15 don't implement it - it is still
better than the 2 implementations using different API, and if it is a
rarely used feature - it is better than attempting to
force the 15 others to implement it, and a good signal that the feature is
not important enough to be part of the standard.
My main concern is with calling this 'experimental' - like we called a lot
of APIs 'alpha' before but still encouraged
users to use in production. Should be treated just like vendor-specific v1
APIs, with the main difference that more than
1 vendor is using the same API.
I used to be very enthusiastic about this proposal - but right now I think
a better approach is to do nothing,
and let each implementation define their own v1 API - perhaps with some
communication and discussion on
a common behavior. Let them try different things, get feedback, wait to see
adoption and convergence - and
only then attempt to standardize. Defining any standard API (or protocol)
by committee, before having real
world experience is never working well.
…On Wed, Jan 29, 2025 at 5:55 PM Rob Scott ***@***.***> wrote:
Yeah this specific proposal doesn't really provide isolation for
experimental *fields*. My previous proposal that did (#3106
<#3106>) was
not particularly popular, so I'm trying to start smaller here.
—
Reply to this email directly, view it on GitHub
<#3497 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAUR2S67EXLO4WRTOHXS432NGBBRAVCNFSM6AAAAABTOYPLEKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMBQGE4TENY>
.
You are receiving this because you commented.Message ID:
<kubernetes-sigs/gateway-api/repo-discussions/3497/comments/12001927@
github.com>
|
Beta Was this translation helpful? Give feedback.
-
|
Not sure how supporting 2 groups is 'orders of magnitude' - I would expect
at worst double, and no worse than what Istio does with HttpRoute and
VirtualService.
All implementation I know have vendor extensions - and I would expect they
would remain supported, not thrown away
even if a subset or variant of the extension is adopted ( and this is
likely to be on a permanent basis for most
implementations - expecting that upstream implements every single features
that any vendor needs is not viable).
Maybe the confusion is 'experiments' ( which should not be used by anyone
in production ) versus features
like FooRoute - that some vendors may adopt and maintain on their own for
their customers. There is
no expectation that a vendor FooRoute will go away and be replaced by an
upstream FooRoute, just like
there was no expectation that VirtualService will be deprecated when
HttpRoute was v1.
Since 'experiments' should never be in production - I am not at all
concerned with their fate, as long as we avoid
the istio confusion between alpha and v1.
The (single or multi-vendor) FooRoute can be v1 and can have more fields or
behave differently - and may
have its own vendor-defined support lifecycle, and since common APIs are
expected to be implementable
by everyone - we can't expect implementations that can support more
features to drop that.
…On Fri, Jan 31, 2025 at 9:26 AM John Howard ***@***.***> wrote:
@mikemorris <https://github.com/mikemorris> the big concern with
splitting groups (from a maintainer POV) is the cost to support 2 groups is
orders of magnitude larger than to support 2 versions of the same group
(since versions are identical).
So I really don't want to get into a world where an implementation has to
deal with that.
The easy way out is... say GW v1.5 promotes FooRoute. So v1.4 had
xFooRoute. v1.5 should remove xFooRoute and add FooRoute (stable).
Implementations supporting v1.5 should *replace* xFooRoute support with
FooRoute support.
tl;dr "within a projects codebase"
—
Reply to this email directly, view it on GitHub
<#3497 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAUR2XLEC7TXQXEIZJ4N5L2NOW5BAVCNFSM6AAAAABTOYPLEKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMBSGA2DKOI>
.
You are receiving this because you commented.Message ID:
<kubernetes-sigs/gateway-api/repo-discussions/3497/comments/12020459@
github.com>
|
Beta Was this translation helpful? Give feedback.
-
|
supporting 2 versions is literally 0 effort so orders of magnitude was incorrect - more like infinite 🙂 I agree vendor Foo and experimental Foo are different, and this proposal is for experimental. The difference being that as (for example) the Istio project there is a ~0% chance I am going to implement GKEFoo and a very high chance I will implement ExperimentalFoo |
Beta Was this translation helpful? Give feedback.
-
|
I get where you are coming from and would agree a lot more if that is what was being proposed - but it doesn't seem to be. |
Beta Was this translation helpful? Give feedback.
-
|
In other words: I read x-k8s.io as "eXtended" - not 'eXperimental'.
A collection of gateway API extensions that have common definition across
few vendors - and are as v1
as the core APIs - but can't be supported by majority/all vendors, or are
too corner cases.
If some of those extensions become common or popular - the same or slightly
different API can be added
to core, of course - and the vendors will have a 2x effort to keep both.
…On Fri, Jan 31, 2025 at 5:14 PM Costin Manolache ***@***.***> wrote:
Not sure how supporting 2 groups is 'orders of magnitude' - I would expect
at worst double, and no worse than what Istio does with HttpRoute and
VirtualService.
All implementation I know have vendor extensions - and I would expect they
would remain supported, not thrown away
even if a subset or variant of the extension is adopted ( and this is
likely to be on a permanent basis for most
implementations - expecting that upstream implements every single features
that any vendor needs is not viable).
Maybe the confusion is 'experiments' ( which should not be used by anyone
in production ) versus features
like FooRoute - that some vendors may adopt and maintain on their own for
their customers. There is
no expectation that a vendor FooRoute will go away and be replaced by an
upstream FooRoute, just like
there was no expectation that VirtualService will be deprecated when
HttpRoute was v1.
Since 'experiments' should never be in production - I am not at all
concerned with their fate, as long as we avoid
the istio confusion between alpha and v1.
The (single or multi-vendor) FooRoute can be v1 and can have more fields
or behave differently - and may
have its own vendor-defined support lifecycle, and since common APIs are
expected to be implementable
by everyone - we can't expect implementations that can support more
features to drop that.
On Fri, Jan 31, 2025 at 9:26 AM John Howard ***@***.***>
wrote:
> @mikemorris <https://github.com/mikemorris> the big concern with
> splitting groups (from a maintainer POV) is the cost to support 2 groups is
> orders of magnitude larger than to support 2 versions of the same group
> (since versions are identical).
>
> So I really don't want to get into a world where an implementation has to
> deal with that.
>
> The easy way out is... say GW v1.5 promotes FooRoute. So v1.4 had
> xFooRoute. v1.5 should remove xFooRoute and add FooRoute (stable).
> Implementations supporting v1.5 should *replace* xFooRoute support with
> FooRoute support.
>
> tl;dr "within a projects codebase"
>
> —
> Reply to this email directly, view it on GitHub
> <#3497 (reply in thread)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAAUR2XLEC7TXQXEIZJ4N5L2NOW5BAVCNFSM6AAAAABTOYPLEKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMBSGA2DKOI>
> .
> You are receiving this because you commented.Message ID:
> <kubernetes-sigs/gateway-api/repo-discussions/3497/comments/12020459@
> github.com>
>
|
Beta Was this translation helpful? Give feedback.
-
|
On Fri, Jan 31, 2025 at 5:23 PM John Howard ***@***.***> wrote:
supporting 2 versions is literally 0 effort so orders of magnitude was incorrect - more like infinite 🙂
I agree vendor Foo and experimental Foo are different, and this proposal is for experimental. The difference being that as (for example) the Istio project there is a ~0% chance I am going to implement GKEFoo and a very high chance I will implement ExperimentalFoo
Right, most vendors will implement APIs defined by another vendor -
and that is the problem,
each vendor defining a XXXFoo API that is slightly different from each
other - all v1 and supported but
subtly different.
Having a common gateway-extensions.io/Foo is far better for users, if
2-3 vendors agree on a common Foo.
From the 'core' API perspective, each single vendor extension and each
multi-vendor extension are 'experimental',
in the sense that there is no clear evidence it is a universal problem
or consensus across all vendors. I think that
is the main purpose (and benefit) of vendor extensions, it allows
vendor to 'experiment' by releasing features
and APIs.
If x-net.io/Foo (v1) was defined - maybe both Istio and GKE and other
could implement it, users can adopt
such a feature with confidence it will be supported as a v1 API, and
when a core Foo API is defined we can
use the lessons learned from x-net.io/Foo and make changes as needed.
Istio implementing ExperimentalFoo - and dealing with migrations and
users who (as usual) adopt
ExperimentalFoo in production ignoring the alpha and experimental
labels - is indeed order of magnitudes
harder than support an x-net.io/FooV1 or an istio.io/FooV1 API long
term and not dealing with any migration.
|
Beta Was this translation helpful? Give feedback.
-
As a follow up to #3106, I'd like to propose a smaller subset of that to address many of the same problems:
Importantly this means that when a new resource graduates to GA, users would need to manually copy their config over from x-k8s.io (alpha) to k8s.io (v1) CRs. Although this is rather onerous, I'd argue that it's preferable to the current experience which can result in confusing error messages when trying to install Gateway API standard channel. This new approach would ensure that it was always safe to install standard channel CRDs. For example, this could allow Kubernetes to include Gateway API standard channel by default (x-ref #3576).
I don't really like 3) here as it means we're not solving the full set of problems that fully separate API groups for experimental channel and standard channel would, but this more limited approach does come with some advantages. Notably, controllers would not need to support both experimental and standard channel API groups at the same time for the same resource. Instead, they could transition to the
k8s.ioAPI group at the same time the API does. If they want to support a broader set of versions, they can support both API groups for a limited period of time, but importantly any Gateway API release would only ever include a single API group (and version) per resource.Note: All mentions of API groups here are just focused on the suffix. In all cases, they would also include the
gateway.networking.prefix they already do.Beta Was this translation helpful? Give feedback.
All reactions