Kubeconfig state unavailable, Terraform defaults to localhost [bug]

[danlester]

Seen most often in the upgrade from 0.3.12 to main (0.4 candidate) of an AWS cluster, Terraform kubernetes provider sometimes tries to access the Kubernetes cluster at localhost instead of the EKS host.

The problem is with these two QHub Terraform modules: kubernetes and kubernetes-initialization.

The kubernetes module creates the cluster itself, and then kubernetes-initialization creates the namespace inside the cluster (plus some secrets).

kubernetes-initialization is configured like this:

provider "kubernetes" {
  host                   = module.kubernetes.credentials.endpoint
  cluster_ca_certificate = module.kubernetes.credentials.cluster_ca_certificate
  token                  = module.kubernetes.credentials.token
}

At the moment, there is a terraform apply step run targeting both these modules at the same time. The problem is that, if the state of the kubernetes module is not known (e.g. not refreshed in the right way at the right time), there is no way we can expect the kubernetes-initialization to have the appropriate provider configured correctly - because that depends on outputs of the earlier module.

We do set kubernetes-initialization to depends_on the kubernetes module, so terraform apply shouldn't try to create the namespace before the cluster exists, for example, but the provider needs to be configured at the start of the whole terraform apply step, otherwise it's as though we never passed through the settings such as host so it defaults to localhost.

The best description of this nuance is probably here.

Anyway, I will try splitting out the two modules into separate steps. That means an extra terraform apply call of course, and it might be better to see if we can instead push kubernetes-initialization into a joint step with the following module (kubernetes-ingress) if we set up appropriate depends_on hierarchies.

But in general, we need a proper multistage definition as per issue #847 .

[costrouc]

@danlester due to terraform being split into stages the kubernetes creation and kubernetes provider are now separate steps. This should no longer be an issue. If there are any issues in 0.4.0+ a new issue should be created.