Use InClusterConfig but allow master IP to be overwritten?

[MrHohn]

Is this a BUG REPORT or FEATURE REQUEST?:
/kind feature

What happened:
When trying to deploy kube-proxy DaemonSet with service account, I found InClusterConfig() not usable for kube-proxy, as this function sets master's IP to kubernetes service's VIP, which depends on kube-proxy itself to set up the proxy rules first...

At this point, in order to use service account on kube-proxy, we have to carry the responsibility of provisioning kube-proxy's kubeconfig at cluster deployment level, like what kubeadm does:

  kubeconfig.conf: |
    apiVersion: v1
    kind: Config
    clusters:
    - cluster:
        certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        server: {{ .MasterEndpoint }}
      name: default
    contexts:
    - context:
        cluster: default
        namespace: default
        user: default
      name: default
    current-context: default
    users:
    - name: default
      user:
        tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token

(Or alternatively we may be able to overwrite the Host field after calling InClusterConfig?)

What you expected to happen:
I would like to have a supported way to generate in-cluster config for components like kube-proxy. Probably something like InClusterProxyConfig(host string)?

How to reproduce it (as minimally and precisely as possible):
See kubernetes/kubernetes#51172 (comment).

cc @luxas @murali-reddy

[lavalamp]

I'm not sure it makes sense to add a special constructor for this case, since kube-proxy is really the only component that should ever have that problem. I would expect overwriting the host field to work, though.

…

On Mon, Aug 28, 2017 at 11:46 AM, Zihong Zheng ***@***.***> wrote: *Is this a BUG REPORT or FEATURE REQUEST?*: /kind feature *What happened*: When trying to deploy kube-proxy DaemonSet with service account, I found InClusterConfig() <https://github.com/kubernetes/client-go/blob/v4.0.0/rest/config.go#L305-L309> not usable for kube-proxy, as this function sets master's IP to kubernetes service's VIP, which depends on kube-proxy itself to set up the proxy rules first... At this point, in order to use service account on kube-proxy, we have to carry the responsibility of provisioning kube-proxy's kubeconfig at cluster deployment level, like what kubeadm does <https://github.com/kubernetes/kubernetes/blob/b8fde17fc2135b6639922c5861db13fc934ac328/cmd/kubeadm/app/phases/addons/proxy/manifests.go#L20-L49> : kubeconfig.conf: | apiVersion: v1 kind: Config clusters: - cluster: certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt server: {{ .MasterEndpoint }} name: default contexts: - context: cluster: default namespace: default user: default name: default current-context: default users: - name: default user: tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token (Or alternatively we may be able to overwrite the Host field <https://github.com/kubernetes/client-go/blob/v4.0.0/rest/config.go#L329> after calling InClusterConfig?) *What you expected to happen*: I would like to have a supported way to generate in-cluster config for components like kube-proxy. Probably something like InClusterProxyConfig(host string)? *How to reproduce it (as minimally and precisely as possible)*: See kubernetes/kubernetes#51172 (comment) <kubernetes/kubernetes#51172 (comment)> . cc @luxas <https://github.com/luxas> @murali-reddy <https://github.com/murali-reddy> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#281>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAngligs7_Js3Q8CipFghYTq5GDny2xqks5scwscgaJpZM4PE6G-> .

[murali-reddy]

@MrHohn thanks for looping me in. If its possible to achive this it would be helpful.

I implemented a service proxy in kube-router. I am sure there will be other implementations of service proxy for Kubernetes. This problem will affect any implemenation of service proxy.

[bjhaid]

is explicitly setting the env var KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT not an adequate solve? We do this today in our deployment for all our incluster configurations as we don't use the in cluster VIP (for other reasons though)

[ericchiang]

You can do this today without modifications to client-go:

func InClusterProxyConfig(host string) (*rest.Config, error) {
	token, err := ioutil.ReadFile("/var/run/secrets/kubernetes.io/serviceaccount/token")
	if err != nil {
		return nil, fmt.Errorf("read service account token: %v", err)
	}
	return &rest.Config{
		Host:            host,
		BearerToken:     string(token),
		TLSClientConfig: rest.TLSClientConfig{
			CAFile: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt",
		},
	}, nil
}

Or even:

func InClusterProxyConfig(host string) (*rest.Config, error) {
	config, err := rest.InClusterConfig()
	if err != nil {
		return nil, err
	}
	// Overwrite discovered in-cluster host.
	config.Host = host
	return config, nil
}

The service account token and CA are documented at https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/#accessing-the-api-from-a-pod

Probably better to just improve the documentation for how to create your own in-cluster clients and use the rest package instead of adding more helpers.

[MrHohn]

Thanks all for the inputs. Overwriting the host field makes sense to me, I will go with that path. Closing this issue.

[murali-reddy]

@MrHohn sorry, i am still not clear. while we can set hostfield in InClusterConfig, from service proxy perspective, how do we know host details to set?

[MrHohn]

@MrHohn sorry, i am still not clear. while we can set hostfield in InClusterConfig, from service proxy perspective, how do we know host details to set?

@murali-reddy IMHO the host details (server ip:port in this case) should still be given by cluster deployment layer instead of deriving from client library. The original purpose for this issue was to avoid generating kubeconfig files in cluster deployment layer, and setting hostfield in InClusterConfig seems to be a viable solution to me.

[murali-reddy]

Agree this is not the scope of the bug @MrHohn i guess the problem of how to build a service proxy that can run as daemon set with service accont is still a chicken-and-egg problem.

[MrHohn]

i guess the problem of how to build a service proxy that can run as daemon set with service accont is still a chicken-and-egg problem.

Agree with that :)