KEP 4770: EndpointSlice Controller Flexibility

[LionelJouin]

• One-line PR description: KEP-4770: Create KEP document (Generic reconciler and disabling controller on particular services)

• Issue link: EndpointSlice controller flexibility #4770

• Other comments:

[k8s-ci-robot]

Welcome @LionelJouin!

It looks like this is your first PR to kubernetes/enhancements 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/enhancements has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @LionelJouin. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[aojea]

we have the label managed-by already on the endpointslice controller and the ipaddress, I think we should be consistent and use the same here

rep -r managed-by staging/src/k8s.io/api
staging/src/k8s.io/api/discovery/v1/well_known_labels.go:       LabelManagedBy = "endpointslice.kubernetes.io/managed-by"
staging/src/k8s.io/api/discovery/v1beta1/well_known_labels.go:  LabelManagedBy = "endpointslice.kubernetes.io/managed-by"
staging/src/k8s.io/api/networking/v1alpha1/well_known_labels.go:        LabelManagedBy = "ipaddress.kubernetes.io/managed-by"
staging/src/k8s.io/api/networking/v1beta1/well_known_labels.go: LabelManagedBy = "ipaddress.kubernetes.io/managed-by"

Also, should we add this label to each Service (if it already does not have the label set) to indicate that is managed by the default endpoint slice controller? I kind of think this will be better

[LionelJouin]

Any name is fine to me. For reference, the service.kubernetes.io/service-proxy-name is not added to the service. As highlighted in #4771 (comment), a default controller name would have to be handled for the kubernetes controller manager.

[aojea]

yes, I added this comment just for opening the debate, I'd like to know if there are more thoughts on this

[robscott]

I think endpointslice.kubernetes.io/managed-by is probably fitting here. I'm not entirely sure whether we want this to apply to Endpoints as well though, if so, that could change what's best here.

[sftim]

This KEP really does need to explain how it interacts with the existing label.

[LionelJouin]

I am not sure what exactly has to be clarified with existing labels. This KEP proposes a label on the Service object, endpointslice.kubernetes.io/managed-by is already explained in the Kubernetes documentation.

[sftim]

Make sure that someone reading this KEP is clear on why endpointslice.kubernetes.io/managed-by isn't right and service.kubernetes.io/service-proxy-name isn't right (bear in mind that at the time of writing the docs say you can set service.kubernetes.io/service-proxy-name on a Service and get a useful outcome).

If a KEP isn't clear:

• we shouldn't implement it (we should clarify, then implement)
• someone might implement the wrong thing through a misunderstanding

Making the KEP clear helps to manage that risk.

[LionelJouin]

Thank you for the explanation, I tried to clarify the differences and how the endpointslice.kubernetes.io/managed-by label is impacted.

[aojea]

This part I can not really understand it, maybe is explained later ...

[robscott]

+1, "Endpoints could be supported for the Kubernetes EndpointSlice Mirroring Controller" is particularly confusing to me.

[aojea]

See https://github.com/kubernetes/enhancements/pull/4771/files#r1685502488 , I think there is 2 things to flesh out here:

1. label name
2. default behavior, do we add a label with the default controller name or we assume no-label means default controller

[LionelJouin]

For reference, kube-proxy has no default controller name, non-labelled services are managed by kube-proxy.

[aojea]

yeah,, but those are decision from the past, if we have to do it again, will we do it the same?
I don't have much feedback from this feature beside using it to workaround cilium conformance tests :)
I think that least we should evaluate the different options and see pros and cons and decide

[aojea]

did a first pass, this KEP is missing some more explanation on the high level details, is too low level, it describe perfectly the changes you want to do in the reconciler logic but is not clear the motivations and how is this planned to be used, some more high level information for the reviewers to understand the changes will help ... an example of how these externals controllers plan to use it will be interesting .... an example of how an external endpoint slice controller consuming this library will be used per example

[LionelJouin]

did a first pass, this KEP is missing some more explanation on the high level details, is too low level, it describe perfectly the changes you want to do in the reconciler logic but is not clear the motivations and how is this planned to be used, some more high level information for the reviewers to understand the changes will help ... an example of how these externals controllers plan to use it will be interesting .... an example of how an external endpoint slice controller consuming this library will be used per example

Thank you for the review. Should I only add high level overview, or should I remove too low level details too? Do we have to provide a way to use the EndpointSlice Reconciler for external EndpointSlice Controller consumers? The example for the usage in the EndpointSlice and EndpointSlice Mirroring Controllers are there already, I was thinking maybe those are sufficient. Also, the usage might change in the future as written in the EndpointSlice Reconciler readme: There are NO compatibility guarantees for this repository, yet. It is in direct support of Kubernetes, so branches will track Kubernetes and be compatible with that repo..

[aojea]

I think is fine to keep it the way you have it, just expand , I'm trying to highlight that without understanding the motivations and the expectations of what your proposing is hard to fully understand all the details

[aojea]

/ok-to-test

[aojea]

I'm not in favor of merging because we are constantly bitten by edge problems with Endpoints and EndpointSlices, more recently kubernetes/kubernetes#127370 , this just adds another permutation to maintain in an already complicated area and there are alternatives, like completely replacing the endpointslice controller on the kube-controller-manager or empty selector.

However, in case we move forward, I expect beta criteria to have at least 2 implementations of these endpointslices controller out of tree in use, adding features is expensive, at least we need to be sure that the cost we are paying is worth it.

[sftim]

@LionelJouin this KEP needs to explain how it interacts with the similar, existing label endpointslice.kubernetes.io/managed-by.

See https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/#management

We could add an equivalent label for Endpoints and even automatically mirror the label when mirroring an Endpoints to a set of EndpointSlices.

[sftim]

@shaneutt please see #4771 (review)

If you agree, please consider cancelling approval.

[sftim]

Is this true? What if you define a Service without a selector and then do your own reconciliation?

[LionelJouin]

Thank you for your comment, I clarified that it is not natively possible while using the Service Selector field (The KEP is about this, the fact to use the service as close as possible to how a user would define it for the Kubernetes default network).
About this topic, there are details in this conversation: #4771 (comment)

[sftim]

We should - I pretty much think must - document the "don't specify a selector, then do your own thing" option. It is an alternative and we shouldn't pretend otherwise.

[LionelJouin]

This is covered by the first alternative Well-known label: Empty Selector field and Use Annotation as Selector, I renamed the alternative so it is more clear.

[sftim]

BTW you can't select on annotations.

[LionelJouin]

you can create your own selector in the annotation, here is an example.

apiVersion: v1
kind: Service
metadata:
  name: service
  annotations:
    selector: "app=a"
spec: {}

[sftim]

Noted

[sftim]

What are the smallest number of changes we can make to allow merging this as a provisional KEP?

[sftim]

This isn't a well-known label.

[LionelJouin]

If you refer to the title, sorry for the confusion, originally, the KEP had 2 proposals (well known label + generic endpointslice reconciler), Well-known Label indicates it is an alternative for the well-known label proposal.

[LionelJouin]

I removed it from the title of the alternative.

[sftim]

Got it.

[sftim]

Try this:

Suggested change

	### Well-known label: Use Dummy Selector
	### Private (per cluster) dummy selector label, custom controller

If you don't mean for the dummy selector to use a cluster-private label, you should revise the explanation.

[LionelJouin]

I am not sure what you mean here. the label in the selector are always within the cluster no?

[sftim]

I'm not sure if you mean

### Empty selector field; pseudo-selector in annotation; no well-known label

This alternative doesn't seem to use any well-known label.

[LionelJouin]

I removed it from the title of the alternative.
Originally, the KEP had 2 proposals (well known label + generic endpointslice reconciler), Well-known Label indicates it is an alternative for the well-known label proposal.

[sftim]

OK, resolved then.

[sftim]

Suggested change

This new well-known label will not have any impact on the other well-known label `endpointslice.kubernetes.io/managed-by`. Both labels can coexist and be used together, ``service.kubernetes.io/endpoint-controller-name` on the Service object while `endpointslice.kubernetes.io/managed-by` on the EndpointSlice object. The label `endpointslice.kubernetes.io/managed-by` is used to indicate the controller or entity that manages the EndpointSlice. In contrast, this new label `service.kubernetes.io/endpoint-controller-name`, is used to disable the default Kubernetes EndpointSlice, EndpointSlice Mirroring and Endpoints controllers for a Service and indicates the controller or entity that manages the EndpointSlice for that particular Service.

This new well-known label will not have any impact on the other well-known label `endpointslice.kubernetes.io/managed-by`. Both labels can coexist and be used together, `service.kubernetes.io/endpoint-controller-name` on the Service object while `endpointslice.kubernetes.io/managed-by` on the EndpointSlice object. The label `endpointslice.kubernetes.io/managed-by` is used to indicate the controller or entity that manages the EndpointSlice(s) for that Service. In contrast, this new label `service.kubernetes.io/endpoint-controller-name`, is used to disable the default Kubernetes EndpointSlice, EndpointSlice Mirroring and Endpoints controllers for a Service and indicates the controller or entity that manages the EndpointSlices for that particular Service.

[LionelJouin]

The label endpointslice.kubernetes.io/managed-by is used to indicate the controller or entity that manages an EndpointSlice. Adding "for that Service" is more confusing to me as the label endpointslice.kubernetes.io/managed-by is applied on an EndpointSlice object, not a Service object.

[sftim]

OK, try this:

This new well-known label will not have any impact on the other well-known label `endpointslice.kubernetes.io/managed-by`. Both labels can coexist and be used together, `service.kubernetes.io/endpoint-controller-name` on the Service object while `endpointslice.kubernetes.io/managed-by` on the EndpointSlice object. The label `endpointslice.kubernetes.io/managed-by` is used to indicate the controller or entity that manages the EndpointSlice(s) - for, example, the endpoint slice controller managing endpoints on behalf of a Service.
In contrast, this new label `service.kubernetes.io/endpoint-controller-name`, is used to disable the default Kubernetes endpoint slice, Endpoints-to-EndpointSlice mirroring and endpoints controllers for a Service; the value of the new label indicates the controller or entity that manages the EndpointSlices for that particular Service.

How does that sound?

[sftim]

Suggested change

	TDB
	TBD

[sftim]

I wonder if we want to test that EndpointSlices created before a Service still work how we want them to (kube-proxy behavior for example), even after that Service is eventually added?

[sftim]

Got it.

[sftim]

Noted

[sftim]

Suggested change

Services without selectors will not get any EndpointSlice objects. Therefore, selecting pods can be done in different ways, for example, via annotation. An annotation will be used in the service to select which pods will be used as backend for this service. For example, [nokia/danm](https://github.com/nokia/danm) uses `danm.k8s.io/selector` (e.g. [DANM service declaration](https://github.com/nokia/danm/blob/v4.3.0/example/svcwatcher_demo/services/internal_lb_svc.yaml#L7)), and [projectcalico/vpp-dataplane](https://github.com/projectcalico/vpp-dataplane) uses `extensions.projectcalico.org/selector` (e.g. [Calico-VPP Multinet services](https://github.com/projectcalico/vpp-dataplane/blob/v3.25.1/docs/multinet.md#multinet-services)). To simplify the user experience, a mutating webhook could read the selector, add them to the annotation and clear them from the specs when the type of service is detected.

Services without selectors will not get any EndpointSlice objects. Therefore, selecting pods can be done in different ways, for example, via annotation. An annotation will be used in the service to select which pods will be used as backend for this service. For example, [nokia/danm](https://github.com/nokia/danm) uses `danm.io/selector`, and [projectcalico/vpp-dataplane](https://github.com/projectcalico/vpp-dataplane) uses `extensions.projectcalico.org/selector` (e.g. [Calico-VPP Multinet services](https://github.com/projectcalico/vpp-dataplane/blob/v3.25.1/docs/multinet.md#multinet-services)). To simplify the user experience, a mutating webhook could read the selector, add them to the annotation and clear them from the specs when the type of service is detected.

[aojea]

/approve cancel

I will try to avoid starting new work without finishing the current one,

• There are two repositories open for multi network open with no activity at all since July https://groups.google.com/g/kubernetes-sig-network/c/8Jfm_Qa-Bl0/m/zYk3knOiAAAJ
• We have a feature in alpha for getting IPs in DRA DRA: Resource Claim Status with possible standardized network interface data #4817 but still no implementation, there is DRA CNI driver that is still WIP https://github.com/kubernetes-sigs/cni-dra-driver/pull/1/files

also this is a nice to have feature, since projects already have a solution to this problem working for a long time, and anything we add in core is something we need to support, and AFAIK there is no open issues related to this, so I would infer that is not much of a problem or the use cases are not enough, the README in this repository is really enlightening https://github.com/k8snetworkplumbingwg/multus-service-archived

In the NPWG call at April 18, we decide that this repository is archived. But this does NOT means Kubernetes service for secondary interfaces is impossible. Here is the reason why we decide to close.
Insufficient use-cases and feedback

Also one of the projects that is mentioned has last commit on Sep 2022 https://github.com/nokia/danm/commits/master/ , latest release 2021, kubernetes 1.21

[aojea]

This project last commit is from Sep 2022 https://github.com/nokia/danm/commits/master/

[sftim]

I think for a ruled-out alternative, referencing something stale is OK. The ecosystem doesn't have to buy in to the thing that we, uh, plan to reject.

[aojea]

I'm trying to gauge the importance of this feature and the demand, it seems there are no OSS implementations of this that are "alive" so I wonder if there are users ... or if there are only private implementations

[MrFreezeex]

Hi! Quick comment about a case in Cilium that this PR might help with. The initial goal of this PR is entirely different to what we would be using this for but we actually also use the endpointslice controller/reconciler code in Cilium clustermesh to sync EndpointSlice from remote clusters and allowing to have a more generic reconciler would certainly help us.

This feature is called "endpointslicesync" (https://docs.cilium.io/en/latest/network/clustermesh/services/#endpointslicesync) and to do that we take the endpoints from remote clusters in our internal format that are pulled from the different clusters and we launch a almost identical EndpointSlice controller to the upstream one/the kubernetes one and fake the informers to masquerade our internal Endpoints data as regular pods and make the controller believe he is in a regular case/cluster while we do some manipulation to actually sync endpoints from remote clusters. All the code to do that is here https://github.com/cilium/cilium/tree/main/pkg/clustermesh/endpointslicesync.

We currently have a fork of the EndpointSlice controller/reconciler with some very small patch to help us do that and more importantly isolate the reconciler code so that it's easier to import in our go mod. The forks are here: https://github.com/cilium/endpointslice-controller & https://github.com/cilium/endpointslice-controller (and yes we should also update those fork to keep up with the kubernetes changes 😅).

This would most likely not remove all the tricks that we pulled of to be able to do that but it could be a first step to allow us to directly use the upstream code without forking it. And this might help future implementation attempting to add support for MCS-API and implementing this part of the spec: https://github.com/kubernetes/enhancements/blob/master/keps/sig-multicluster/1645-multi-cluster-services-api/README.md#using-endpointslice-objects-to-track-endpoints. While it would help us I entirely understand this might be a niche use case and you wouldn't want to complexify further the EndpointSlice controller for it.

[aojea]

: https://github.com/kubernetes/enhancements/blob/master/keps/sig-multicluster/1645-multi-cluster-services-api/README.md#using-endpointslice-objects-to-track-endpoints. While it would help us I entirely understand this might be a niche use case and you wouldn't want to complexify further the EndpointSlice controller for it.

We can support a community version of an endpoint slice controller out of tree like kubernetes-sigs/endpointslice-controller so we spread the load, I'm very much on favor of that ...
My "moving intree experience" is the opposite, what usually happens is that people leaves once it merges and maintainers ends dealing with the problem ... a problem that only solves some specific use cases on some specific projects, at least is what it seems today ... if this turns out to be wrong and there is wide adoption we can always put it intree later ... "yes is forever, no is temporary"

[sftim]

/magic-emoji-reaction literally-this

😉