Once upon a time, I had one server, hosted on Kimsufi. It was full of ugly docker-compose files, nothing was automated, it was just plain ol' ssh and that was it.
In 2017, while studying in Poland, I discovered Scaleway, and more generally the cloud. So I ending up migrating everything on some VC1S servers, still using ssh.
2 years later, I started working at Scaleway on Kapsule, their managed Kubernetes solution. Since then, I've always thought of migrating everything to Kubernetes, but was always too lazy to set it up 😅. I still managed to migrate from the VC1S to some Dedibox servers, but again, still using ssh and docker-compose. Nothing was automated, it was a real mess!
Though at work, I was trying to automate everything, my personal setup was not part of it!
So 2021, new year and all that, it was THE year to finally automate everyhting. Disclaimer: even though I'd like a home cluster, I've got no place to host it in my flat yet!
After some reflection, and testing, I ended up choosing baremetal Hetzner servers. Hence my old setup.
Now I have switched to a single node install.
I've chosen to use three servers, acting as worker and control plane. They are linked with a vSwitch, everything binding to the private IP, except SSH.
I've written sone basic Ansible roles, in order to set up the Kubernetes cluster. Well not so basic, it supports a Kubernetes version rolling upgrade 😄!
I'm using Tailscale in order to have acces to the Kubernetes API server. For public access, I'm using MetalLB with a public subnet routed directly in the vSwitch. With the help of Cilium and Direct Server Return (DSR) I'm able to get the real client IP directly into my pods (very useful for the mail server).
For the storage, I'm using Rook, with direct access the drives (yeah I just dropped the RAID!), wich allows me to get Block, and Filesystems storage for my pods.
I'm then using cert-manager and ExternalDNS, both using my Cloudflare account to manage TLS certificate, and DNS.
Regarding the monitoring, I still have a free student Datadog account, so why not use it! (If someone at Datadog reads this, please don't drop that 😄, if you do, I'll switch to a classic Prometheus, Grafana and Loki setup!)
As for the ingress, I've chosen Contour since I've grown kind of fond of Envoy!
I'm also using a (still) local fork of the Hetzner cloud controller manager, to get both the ExternalIP and (vSwitch) InternalIP of my nodes.
I was using the Sealed Secrets, but I switched to sops before writing this.
Finally, for the automation, I was using Flux with a private git repo. Now this will still be with Flux, but in public repo, and a lot of automation taken from here and there!
Most of the git automation here is taken from the awsome @onedr0p and his home-cluster repo and the more widely k8s-at-home community. Kudos to him and the community 🎉!