diff --git a/doc/.wordlist.txt b/doc/.wordlist.txt index 3981e70cc70..06755106f84 100644 --- a/doc/.wordlist.txt +++ b/doc/.wordlist.txt @@ -20,6 +20,7 @@ backported balancer balancers benchmarking +BFD BGP bibi BitLocker @@ -71,6 +72,7 @@ Ebit eBPF ECDHE ECDSA +ECMP EDK EiB Eibit diff --git a/doc/clustering.md b/doc/clustering.md index e74e9199249..b8354cac79e 100644 --- a/doc/clustering.md +++ b/doc/clustering.md @@ -6,11 +6,12 @@ explanation/clustering.md Form a cluster +Access a cluster Manage a cluster Recover a cluster +Manage cluster groups Manage instances Configure storage Configure networks -Set up cluster groups reference/cluster_member_config ``` diff --git a/doc/howto/cluster_access.md b/doc/howto/cluster_access.md new file mode 100644 index 00000000000..7c486c7825f --- /dev/null +++ b/doc/howto/cluster_access.md @@ -0,0 +1,91 @@ +(cluster-access)= +# Accessing a cluster +An Incus cluster generally behaves much like a standalone Incus server. + +A client can talk to any of the servers within a cluster and will get an +identical experience. Requests can be directed at a specific server +through the API and doesn't need a direct connection to that server. + +```{note} +Targeting a specific server is done through `?target=SERVER` at the API level or `--target` at the CLI level). +``` + +The cluster uses a single client facing TLS certificate for all servers, +this makes it easier to expose a valid HTTPS endpoint to clients, +avoiding having to manually check fingerprints. + +You can use `incus cluster update-certificate` to load your own +cluster-wide TLS certificate, or you can use ACME to automatically issue +and deploy a certificate across the cluster (see {ref}`authentication-server-certificate`). + +## Authentication +### HTTPS with TLS +The default authentication method when dealing with a remote Incus cluster. + +This works fine in a cluster, but may cause some issues with some +proxies and load-balancers that want to establish their own TLS +connection to the cluster. + +See {ref}`authentication-tls-certs` for details. + +### HTTPS with OpenID Connect (OIDC) +OpenID Connect authentication on Incus requires an external OpenID +Identity Provider but then has the advantage of offering fine grained +authentication to a cluster, making managing and auditing access easy. + +For OpenID Connect to work properly, the cluster will need to have a DNS +record, a valid certificate and be able to reach the OpenID Identity +Provider. + +See {ref}`authentication-openid` for details. + +### Local access +You can also interact with a cluster by connecting to any of the +clustered servers and talking to the local Incus daemon running on that +server. + +## High availability +To provide a highly available Incus API on a cluster, you need to have +client requests always make it to at least one responsive server. + +Here are a few common ways to handle it. + +### DNS round-robin +DNS is a very easy way to balance API traffic over multiple servers. +Simply create a DNS record with an `A` or `AAAA` for each server in the cluster. + +While this is trivial to put in place, it will only properly handle falling back to another server if the server quickly rejects the connection. +Any stuck server may cause significant delays for some clients as they'll need to wait for a full connection timeout before another server is contacted. + +### External load-balancer +A reasonably easy solution is to run a load-balancer, either a self-hosted one like `haproxy` or one provide by your existing network or cloud infrastructure. + +Those load-balancers can often monitor service health and only send requests to servers that are currently responsive. +Incus supports the `haproxy` proxy protocol headers so the original client IP address is reported in log and audit messages. + +```{note} +TLS client certificate authentication only works with load-balancers that act at the TCP level. + +Load-balancers which terminate the TLS session and then establish their own to Incus can only be used with OIDC authentication. +``` + +### Floating IP address +It's possible to use Incus with an additional floating IP, effectively a virtual IP address which is only live on one of the servers. +This centralizes all client API traffic to that single server but may be easier to manage in some environments. + +For that you'll need to make sure that all servers are configured to listen on all interfaces (e.g. `core.https_address=:8443`) +and then make use of a local firewall to only allow external clients to connect to the virtual IP address. + +Common solutions to handle a virtual IP address are `VRRP` (through something like `frr`) and `corosync/pacemaker`. + +### ECMP +For those running a full L3 network infrastructure with BGP to each individual host, it's possible to advertise an IP address for use for Incus client traffic. + +This IP address would be added to all servers in their network configuration (as a `/32` for IPv4 or `/128` for IPv6) and then advertised to their router. +This will result in the router having an equal cost route for the IP address to all servers in the cluster (ECMP). + +Traffic will then get balanced between all servers and as soon as a server goes down, its route will go away and traffic will head to the remaining servers. + +```{note} +To minimize fallback delay, one can make use of BFD alongside BGP to get sub-1s fallback time. +```