Support Connection to ResourceInterpretWebhook without DNS Services

[lxtywypc]

What would you like to be added:
In the code of resourceinterpreter, add the logic of building a new serviceLister to initialize a new ClusterIPServiceResolver instead of DefaultServiceResolver.

Why is this needed:
If we deploy karmada with Default dnsPolicy or deploy karmada in physical machine/virtual machine directly, and there is no coreDNS or other DNS services, it would not connect to resource-interpret-webhooks successfully.

[XiShanYongYe-Chang]

Hi @lxtywypc, thanks for your contribution! Have you started using Karmada?

[lxtywypc]

Hi @lxtywypc, thanks for your contribution! Have you started using Karmada?

Yes, and I'm working for Trip.com now. We met this problem when we tried to build a customized resource-interpreter-webhook. In our opinion, karmada components should work independently without relying other DNS service.

[RainbowMango]

Can someone remind me which service the NewDefaultServiceResolver resolves?

karmada/pkg/resourceinterpreter/customizedinterpreter/customized.go

Line 49 in eda7814

cm.SetServiceResolver(webhookutil.NewDefaultServiceResolver())

[XiShanYongYe-Chang]

Hi @lxtywypc, is the resolved address configured for network mounting in MutatingWebhookConfiguration and ValidatingWebhookConfiguration?

Like this:
https://github.com/karmada-io/karmada/blob/master/artifacts/deploy/webhook-configuration.yaml#L16

[lxtywypc]

Can someone remind me which service the NewDefaultServiceResolver resolves?

karmada/pkg/resourceinterpreter/customizedinterpreter/customized.go

Line 49 in eda7814

cm.SetServiceResolver(webhookutil.NewDefaultServiceResolver())

Hi @lxtywypc, is the resolved address configured for network mounting in MutatingWebhookConfiguration and ValidatingWebhookConfiguration?

Like this: https://github.com/karmada-io/karmada/blob/master/artifacts/deploy/webhook-configuration.yaml#L16

It resolves the service decalred in ResourceInterpreterWebhookConfiguration. Karmada-controller-manager would connect to customized resource-interpreter-webhook by the resolved result.

Like this:
https://github.com/karmada-io/karmada/blob/master/examples/customresourceinterpreter/webhook-configuration.yaml#L14
But not "url", it should be "service".

karmada/vendor/k8s.io/api/admissionregistration/v1/types.go

Line 534 in c34c443

Service *ServiceReference `json:"service,omitempty" protobuf:"bytes,1,opt,name=service"`

DefaultServiceResolver could only resolve it to a service URL like https://{{service-name}}.{{service-namespace}}.svc:443, but it couldn't work if there was no coreDNS or other DNS service. It would work independently if the service was resolved to ClusterIP like https://{{service-cluster-ip}}:443.

[RainbowMango]

But not "url", it should be "service".

Do you mean you specify the webhook by a service? like this:

    clientConfig:
      service:
        namespace: foo
        name: bar

And the ServiceResolver only takes effect for this case(specify webhook by a service), am I right?

[lxtywypc]

But not "url", it should be "service".

Do you mean you specify the webhook by a service? like this:

    clientConfig:
      service:
        namespace: foo
        name: bar

And the ServiceResolver only takes effect for this case(specify webhook by a service), am I right?

I think so it does.

[RainbowMango]

OK. Thanks.

So, the DefaultServiceResolver resolves a service to a URL with DNS name, like https://ross.andromeda.svc:443. And the ClusterIPServiceResolver resolves a service to a URL according to service type:

• ClusterIP/LoadBalancer/NodePort: net.JoinHostPort(svc.Spec.ClusterIP, fmt.Sprintf("%d", svcPort.Port))
• ExternalName: net.JoinHostPort(svc.Spec.ExternalName, fmt.Sprintf("%d", port))

Seems ClusterIPServiceResolver doesn't rely on a DNS resolver.
I have two questions for now:

1. How does the kube-webhook handle it, I mean which resolver it uses?
2. For this case, no coreDNS, if we can use the URL way instead of the service?

[lxtywypc]

OK. Thanks.

So, the DefaultServiceResolver resolves a service to a URL with DNS name, like https://ross.andromeda.svc:443. And the ClusterIPServiceResolver resolves a service to a URL according to service type:

• ClusterIP/LoadBalancer/NodePort: net.JoinHostPort(svc.Spec.ClusterIP, fmt.Sprintf("%d", svcPort.Port))
• ExternalName: net.JoinHostPort(svc.Spec.ExternalName, fmt.Sprintf("%d", port))

Seems ClusterIPServiceResolver doesn't rely on a DNS resolver. I have two questions for now:

1. How does the kube-webhook handle it, I mean which resolver it uses?
2. For this case, no coreDNS, if we can use the URL way instead of the service?

I'm sorry for replying late. It took a little long to read the code of kube-apiserver.

1. kube-webhook is accessed by kube-apiserver. It seems that kube-apiserver register a new Webhook first whose clientManager was built with DefaultServiceResolver
   https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating/plugin.go#L34
   and then initialize the webhook with the LoopbackServiceResolver whose core is EndpointServiceResolver if enable aggregator routing and is ClusterIPServiceResolver otherwise.
   https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-apiserver/app/server.go#L461#L473
   https://github.com/kubernetes/kubernetes/blob/master/pkg/kubeapiserver/admission/config.go#L52
   https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/server/options/admission.go#L158

2. We deploy the ResourceInterpretWebhook and its service in cluster by CI. If we use the URL way, we have to deploy the service first, then login the cluster to get service IP manually, fill the configuration and then deploy it. If the service is deleted unexpectly, or its IP is changed, we have to do as above again. And if we don't want to operation cluster manually too much...
   In summary, we think it is better to use the Service way to build the configuration if the service is deployed in cluster.

[XiShanYongYe-Chang]

Hi @lxtywypc,

For the second point, can I understand it this way: If there is no DNS server in the cluster, the service address in the URL, like https://ross.andromeda.svc:443, cannot be automatically resolved. In this case, the Service configuration mode is recommended.

[lxtywypc]

Hi @lxtywypc,

For the second point, can I understand it this way: If there is no DNS server in the cluster, the service address in the URL, like https://ross.andromeda.svc:443, cannot be automatically resolved. In this case, the Service configuration mode is recommended.

Yes, I think it is better.

[RainbowMango]

/assign @lxtywypc