k3s agent cannot connect to server via proxy when flag --disable-apiserver-lb enabled

[htrongnguyen94]

I want to test connect k3s agent via Proxy and with flag --disable-apiserver-lb, because in actual environment my k3s-server is behind an external-loadbalancer
I created 2 VM for k3s-server and k3s-client.
k3s-server start cmd: k3s server --agent-token mytoken --tls-san <server-public-ip> --node-external-ip <server-public-ip>

In k3s-client VM, I export the [HTTP|HTTPS]_PROXY env, NO_PROXY=localhost,127.0.0.1
start cmd: k3s agent --server https://<server-public-ip>:6443 --token mytoken --disable-apiserver-lb
Result log

INFO[0000] Starting k3s agent v1.29.6+k3s1 (83ae095)
ERRO[0010] failed to get CA certs: Get "https://<server-public-ip>:6443/cacerts": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

I try with curl and it's OK

curl -kI https://<server-public-ip>:6443/cacerts
HTTP/1.1 200 Connection established
HTTP/2 200
content-type: text/plain
content-length: 570

I also try to start k3s server/agent by using systemd service but it had the same result
I didn't found any issue reported this
So I wonder if K3S support connect through Proxy with --disable-apiserver-lb or not ?

[brandond]

Are you sure properly set the proxy env vars for the K3s service? You should put them in etc/sysconfig/k3s or /etc/systemd/system/k3s.service.env, just setting them in your shell won't affect services managed by systemd.

I can't say we've tested it with a proxy in between nodes. It is not generally expected that there would be a HTTP proxy between server and agent or the registration endpoint; there needs to be connectivity between all cluster nodes in order for the CNI overlay network (vxlan) to function properly so that pods and services are able to communicate.

brandond@dev01:~$ kubectl get node -o wide
NAME           STATUS   ROLES                  AGE     VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION   CONTAINER-RUNTIME
k3s-agent-1    Ready    <none>                 2m38s   v1.29.6+k3s1   172.17.0.9    <none>        K3s v1.29.6+k3s1   6.6.0-1001-aws   containerd://1.7.17-k3s1
k3s-server-1   Ready    control-plane,master   2m48s   v1.29.6+k3s1   172.17.0.8    <none>        K3s v1.29.6+k3s1   6.6.0-1001-aws   containerd://1.7.17-k3s1

brandond@dev01:~$ kubectl get node -o yaml | grep node-args
      k3s.io/node-args: '["agent","--token","********","--disable-apiserver-lb","--server","https://172.17.0.8:6443"]'
      k3s.io/node-args: '["server","--token","********"]'

[brandond]

The http.Client we construct does not enable reading proxy config from the environment, since it's supposed to be talking to the local apiserver load-balancer endpoint and we wouldn't want to go through a proxy for that... so I suspect that no, it will not work through a proxy.

k3s/pkg/clientaccess/token.go

Lines 260 to 266 in 40eda6a

	client := &http.Client{
	Timeout: defaultClientTimeout,
	Transport: &http.Transport{
	DisableKeepAlives: true,
	TLSClientConfig: tlsConfig,
	},
	}

The apiserver load-balancer does support use of a proxy, if you enable it via an env var:

k3s/pkg/agent/loadbalancer/servers.go

Lines 26 to 33 in 40eda6a

	// SetHTTPProxy configures a proxy-enabled dialer to be used for all loadbalancer connections,
	// if the agent has been configured to allow use of a HTTP proxy, and the environment has been configured
	// to indicate use of a HTTP proxy for the server URL.
	func SetHTTPProxy(address string) error {
	// Check if env variable for proxy is set
	if useProxy, _ := strconv.ParseBool(os.Getenv(version.ProgramUpper + "_AGENT_HTTP_PROXY_ALLOWED")); !useProxy || address == "" {
	return nil
	}

I think the expectation was, if you're trying to enable direct communication, to the point where you disable the agent's apiserver load-balancer, you wouldn't be sticking a proxy in the way of things.

We could perhaps consider allowing use of a proxy via that same env var, when the agent's apiserver load-balancer is disabled.

[htrongnguyen94]

Thank you for your details answer.

[brandond]

My recommendation would be to skip using --disable-apiserver-lb, and set K3S_AGENT_HTTP_PROXY_ALLOWED=1 in the k3s env file.

Note that this still won't fix the CNI; there is no way flannel is ever going to work though a proxy.