The Hetzner Cloud cloud-controller-manager integrates your Kubernetes cluster with the Hetzner Cloud & Robot APIs.
- Node:
- Updates your
Nodeobjects with information about the server from the Cloud & Robot API. - Instance Type, Location, Datacenter, Server ID, IPs.
- Updates your
- Node Lifecycle:
- Cleans up stale
Nodeobjects when the server is deleted in the API.
- Cleans up stale
- Routes (if enabled):
- Routes traffic to the pods through Hetzner Cloud Networks. Removes one layer of indirection in CNIs that support this.
- Load Balancer:
- Watches Services with
type: LoadBalancerand creates Hetzner Cloud Load Balancers for them, adds Kubernetes Nodes as targets for the Load Balancer.
- Watches Services with
Read more about cloud controllers in the Kubernetes documentation.
apiVersion: v1
kind: Node
metadata:
labels:
node.kubernetes.io/instance-type: cx22
topology.kubernetes.io/region: fsn1
topology.kubernetes.io/zone: fsn1-dc8
instance.hetzner.cloud/provided-by: cloud
name: node
spec:
podCIDR: 10.244.0.0/24
providerID: hcloud://123456 # <-- Hetzner Cloud Server ID
status:
addresses:
- address: node
type: Hostname
- address: 1.2.3.4 # <-- Hetzner Cloud Server public ipv4
type: ExternalIPThis deployment example uses kubeadm to bootstrap an Kubernetes
cluster, with flannel as overlay
network agent. Feel free to adapt the steps to your preferred method of
installing Kubernetes.
These deployment instructions are designed to guide with the
installation of the hcloud-cloud-controller-manager and are by no
means an in depth tutorial of setting up Kubernetes clusters.
Previous knowledge about the involved components is required.
Please refer to the kubeadm cluster creation guide, which these instructions are meant to augment and the kubeadm documentation.
-
The cloud controller manager adds the labels when a node is added to the cluster. For current Kubernetes versions, this means we have to add the
--cloud-provider=externalflag to thekubelet. How you do this depends on your Kubernetes distribution. Withkubeadmyou can either set it in the kubeadm config (nodeRegistration.kubeletExtraArgs) or through a systemd drop-in unit/etc/systemd/system/kubelet.service.d/20-hcloud.conf:[Service] Environment="KUBELET_EXTRA_ARGS=--cloud-provider=external"
Note: the
--cloud-providerflag is deprecated since K8S 1.19. You will see a log message regarding this. For now (v1.31) it is still required. -
Now the control plane can be initialized:
sudo kubeadm init --pod-network-cidr=10.244.0.0/16
-
Configure kubectl to connect to the kube-apiserver:
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
-
Deploy the flannel CNI plugin:
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
-
Patch the flannel deployment to tolerate the
uninitializedtaint:kubectl -n kube-system patch ds kube-flannel-ds --type json -p '[{"op":"add","path":"/spec/template/spec/tolerations/-","value":{"key":"node.cloudprovider.kubernetes.io/uninitialized","value":"true","effect":"NoSchedule"}}]' -
Create a secret containing your Hetzner Cloud API token.
kubectl -n kube-system create secret generic hcloud --from-literal=token=<hcloud API token>
-
Deploy
hcloud-cloud-controller-managerUsing Helm (recommended):
helm repo add hcloud https://charts.hetzner.cloud helm repo update hcloud helm install hccm hcloud/hcloud-cloud-controller-manager -n kube-systemSee the Helm chart README for more info.
Legacy installation method:
kubectl apply -f https://github.com/hetznercloud/hcloud-cloud-controller-manager/releases/latest/download/ccm.yaml
When you use the Cloud Controller Manager with networks support, the CCM is in favor of allocating the IPs (& setup the routing) (Docs: https://kubernetes.io/docs/concepts/architecture/cloud-controller/#route-controller). The CNI plugin you use needs to support this k8s native functionality (Cilium does it, I don't know about Calico & WeaveNet), so basically you use the Hetzner Cloud Networks as the underlying networking stack.
When you use the CCM without Networks support it just disables the RouteController part, all other parts work completely the same. Then just the CNI is in charge of making all the networking stack things. Using the CCM with Networks support has the benefit that your node is connected to a private network so the node doesn't need to encrypt the connections and you have a bit less operational overhead as you don't need to manage the Network.
If you want to use the Hetzner Cloud Networks Feature, head over to
the Deployment with Networks support
documentation.
If you manage the network yourself it might still be required to let the CCM know about private networks. You can do this by adding the environment variable with the network name/ID in the CCM deployment.
env:
- name: HCLOUD_NETWORK
valueFrom:
secretKeyRef:
name: hcloud
key: network
You also need to add the network name/ID to the
secret: kubectl -n kube-system create secret generic hcloud --from-literal=token=<hcloud API token> --from-literal=network=<hcloud Network_ID_or_Name>
.
If kube-proxy is run in IPVS mode, the Service manifest needs to have the
annotation load-balancer.hetzner.cloud/hostname where the FQDN resolves to the HCloud LoadBalancer IP.
See #212
We aim to support the latest three versions of Kubernetes. When a Kubernetes version is marked as End Of Life, we will stop support for it and remove the version from our CI tests. This does not necessarily mean that the Cloud Controller Manager does not still work with this version. We will not fix bugs related only to an unsupported version.
Current Kubernetes Releases: https://kubernetes.io/releases/
To set up a development environment, make sure you installed the following tools:
- Configure a
HCLOUD_TOKENin your shell session.
Warning
The development environment runs on Hetzner Cloud servers which will induce costs.
- Deploy the development cluster:
make -C dev up- Load the generated configuration to access the development cluster:
source dev/files/env.sh- Check that the development cluster is healthy:
kubectl get nodes -o wide- Start developing hcloud-cloud-controller-manager in the development cluster:
skaffold devOn code change, skaffold will rebuild the image, redeploy it and print all logs.
make -C dev downTo run the unit tests, make sure you installed the following tools:
- Run the following command to run the unit tests:
go test ./...Before running the e2e tests, make sure you followed the Setup a development environment steps.
- Run the kubernetes e2e tests using the following command:
source dev/files/env.sh
go test ./tests/e2e -tags e2e -vIf you want to work on the Robot support, you need to make some changes to the above setup.
This requires that you have a Robot Server in the same account you use for the development. The server needs to be setup with the Ansible Playbook dev/robot/install.yml and configured in dev/robot/install.yml.
- Set these environment variables:
export ROBOT_ENABLED=true
export ROBOT_USER=<Your Robot User>
export ROBOT_PASSWORD=<Your Robot Password>-
Continue with the environment setup until you reach the
skaffoldstep. Runskaffold dev --profile=robotinstead. -
We have another suite of tests for Robot. You can run these with:
go test ./tests/e2e -tags e2e,robot -vApache License, Version 2.0