Setup High Availability with Vault Operator

[LukasKnuth]

Hey, we have the following issue in our cluster:

1. A Node dies on which the single Vault instance managed by the operator runs
2. Kubernetes notices and restarts all Pods that where on that node somewhere else
3. Many application Pods fail to start because they depend on secrets injected by the Vault Webhook (which can't find it's Vault instance)
4. The vault instance eventually comes back
5. All applications that where started before Vault was back are stuck in CrashLoop states

I see two issues with our current configuration:

We only have a single instance of Vault running in our cluster. If the Node it runs on dies, we have this problem.

I can't find any examples in the Vault Operator repo/docs on how to run multiple instances. I know there is spec.size which i can set to a higher number but I have no idea if that is all thats needed.

For example, Do these instances find each other? Is their Affinity configured so that they aren't scheduled onto the same Node?

---

If the Bank Vaults Webhook is not successful in injecting the secrets from Vault, it starts the Pod anyways. As far as I can tell, once the Pod is created, the Webhook doesn't run again to try and fetch the secrets again. This leaves these Pods dead until their Deployment is manually restarted.

I can't find any Annotations to configure the Webhook to change this behavior. Is it possible to have this automatically retry periodically?

[rljohnsn]

What storage do you have configured? We are quite satisfied with the HA setup using Integrated storage (Raft). Here is a sample config block. (AWS, EKS, KMS) Any pod killed one of the others will auto take over. We can expand to any number of replicas and they will auto join the cluster. Access to secrets not affected by pod restarts. The resulting statefulset spreads the pods on different hosts by default.

For secrets access we are using External Secrets Operator

[LukasKnuth]

We are using GCS as the storage backend. I found the examples in the /deploy/examples folder of the vault-operator folder, mainly cr-gcs-ha.yaml.

On top of that, I set the following affinity rule to not schedule two pods onto the same Node.

affinity:
  # Don't schedule multiple Vault Pods onto the same Node
  podAntiAffinity:
    # NOTE: Change this to "preferredXXX" if scheduling causes issues.
    requiredDuringSchedulingIgnoredDuringExecution:
      - topologyKey: "kubernetes.io/hostname"
        labelSelector:
          matchLabels:
            # Default label set by Vault Operator
            "app.kubernetes.io/name": "vault"

Lastly, the official docs explain how high availability works for Vault, which is basically what @rljohnsn also said: Other instances are waiting until the primary instance goes down, the leadership election is handled through the storage backend (which must support HA):

Vault supports a multi-server mode for high availability. This mode protects against outages by running multiple Vault servers. High availability mode is automatically enabled when using a data store that supports it.

[...]

To be highly available, one of the Vault server nodes grabs a lock within the data store. The successful server node then becomes the active node; all other nodes become standby nodes. At this point, if the standby nodes receive a request, they will either forward the request or redirect the client depending on the current configuration and state of the cluster -- see the sections below for details. Due to this architecture, HA does not enable increased scalability. In general, the bottleneck of Vault is the data store itself, not Vault core.

That was it :)