Add alpha support for local libvirt (cl only) #266
Conversation
There was a problem hiding this comment.
Hey Casey, thanks for the unexpected effort. This may idle for a while however.
There are other topics I want to address before supporting a new platform (as exciting as that is). I'll also need to do plenty of messing about with https://github.com/dmacvicar/terraform-provider-libvirt since I last gave it a shot.
| Make a note of the absolute path to this image, you'll need it later. | ||
|
|
||
|
|
||
| ## DNS |
There was a problem hiding this comment.
Seems overly complicated. I'm not keen to have users setup local split horizon or to answer questions about it when its inevitably "not working" for someone.
docs/cl/libvirt.md
Outdated
|
|
||
| cluster_name = "hestia" | ||
| base_image_path = "/home/user/coreos.img" | ||
| domain = "hestia.k8s" |
There was a problem hiding this comment.
Ah, but Hestia is the goddess of the home. Because it's local development. eh? eh?
docs/cl/libvirt.md
Outdated
| domain = "hestia.k8s" | ||
|
|
||
| controller_names = ["node1", "node2"] | ||
| controller_ips = ["192.168.120.10", "192.168.120.11"] |
There was a problem hiding this comment.
No IPs. Users should not be permitted to hardcode IPs.
There was a problem hiding this comment.
Having stable IPs for the controllers makes some things easier for DNS. They could be autogenerated from the node_ip_pool. The only downside to this is that they could change if the controller count changes.
docs/cl/libvirt.md
Outdated
|
|
||
| In this tutorial, boot and provision a Kubernetes v1.11.0 using a local libvirt instance. | ||
|
|
||
| The libvirt archirecture is meant for development and testing, rather than long-term use. It does not support load-balancing between multiple API servers, for example. |
There was a problem hiding this comment.
Phrasing is crucial here. Development/testing can't be grounds for the architecture or conventions to differ between Typhoon's platforms. We can say its only suitable for local user though, that's fine.
All Typhoon clusters should allow multi-master setups. This can be treated just like bare-metal, where load balancing is left to how the end user chooses to resolve / balance across masters.
docs/cl/libvirt.md
Outdated
| sudo dnf install libvirt-client qemu-img | ||
| ``` | ||
|
|
||
| ## CoreOS Base image |
There was a problem hiding this comment.
Typhoon consistently refers to these as "Container Linux Images". It accomodates Flatcar and future forks.
| default = "default" | ||
| } | ||
|
|
||
| variable "libvirt_endpoint" { |
| default = "br42" | ||
| } | ||
|
|
||
| variable "libvirt_storage_pool" { |
| default = "qemu:///system" | ||
| } | ||
|
|
||
| variable "dns_server" { |
There was a problem hiding this comment.
We do need something like this; libvirt requires an upstream DNS server.
|
|
||
| # optional | ||
|
|
||
| variable "cluster_domain_suffix" { |
There was a problem hiding this comment.
We can remove this too. Its only justified use was in real production clusters.
There was a problem hiding this comment.
Do you mind if I keep it? Just to limit divergence from other platforms.
| description = "The domain name to use for the cluster (e.g. k8s.example.com)" | ||
| } | ||
|
|
||
| variable "k8s_hostname" { |
|
Yeah, I threw this together quickly to scratch a personal itch and figured I'd throw over a PR. It's cool if it stews for a bit. DNS is tricky. The ideal is to be able to spin up a cluster without overly complicated external configuration, like a load-balancer or external DNS records. We're also limited by libvirt and terraform-provider-libvirt - we can't set up CNAMEs or add multiple IPs to a single name. The absolute requirements are:
The nice-to-haves are:
Solutions I can think of:
I can't immediately think of a way to avoid split-horizon. If you want world-resolvable domain names, then you should probably be using the bare-metal platform anyways? |
|
Okay, I managed to cut down on most of the DNS insanity. Here's the solution I came up with:
We still have to set a single domain for all machine hostnames, since the etcd servers rely on that. |
This modifies the bare-metal platform slightly for direct libvirt provisioning. It uses terraform-provider-libvirt, and can bring up a test cluster in under 4 minutes.
|
@squeed ciao which network option you will need for this project ? speaking from libvirt-terraform perspective. |
|
@MalloZup the big thing that would make this a lot easier would be the ability to manage domain names via terraform-libvirt. For example, right now I have to shell out to virsh: https://github.com/poseidon/typhoon/pull/266/files#diff-32eed82232b8dd46a8490b50515327ebR28 |
| domain = "${var.machine_domain}" | ||
| addresses = ["${var.node_ip_pool}"] | ||
|
|
||
| dns_forwarder { |
There was a problem hiding this comment.
need to update for terraform-provider-libvirt 0.5.1:
dns {
forwarders = [
{
address = "${var.dns_server}"
}
]
}
|
I've ported the latest Typhoon (Kube 1.13.2) to libVirt / CoreOS. It works great! I used a libVirt NAT network for the private network. For the bastion-host I used a bridge network to allow access from LAN. At first I've tried to run a HTTP/HTTPS-Proxy to route external request to bastion host. But this HTTP_PROXY/NO_PROXY never worked with Kubernetes control plane. So the best was to use a gateway for this. I used Terraform 0.11.11 and libVirt provider 0.5.1. The base terraform looks like similar to this: module "libvirt-example" {
source = "//libvirt/container-linux/kubernetes"
cluster_name = "mycluster"
os_image = "images/coreos_production_qemu_image.img"
controller_names = ["master"]
worker_names = [ "worker1", "worker2" ]
machine_domain = "company.k8s"
k8s_domain_name = "master.company.k8s"
machine_public_domain = "company-public.k8s"
ssh_authorized_key = "..."
ssh_private_key = "${file("~/.ssh/id_rsa")}"
asset_dir = "/assets"
}Sorry, I currently don't have a public repository of it. |
|
I'm unlikely to expand support to libvirt, just don't use it enough among all the regular clusters |
This modifies the bare-metal platform slightly for direct libvirt provisioning. It uses terraform-provider-libvirt, and can bring up a test cluster in under 4 minutes.
Testing
Run through the Libvirt howto. Works on my machine :-).