Failed to initialize aws-auth configmap with only self managed node group

[bohdantverdyi]

When module eks uses only self managed group, it means that no aws-auth exists (only managed-group can generate automatically) .

Nodes can't connect to EKS, when no aws-auth exists in cluster, finally job can't execute becuase no nodes connected to cluster (jobs can't connect becuase aws-auth not exists )

Solution would be to add new option to create config-map with aws-auth before job execution .

[bohdantverdyi]

Better idea to add option to create only config map, without jobs.

[aidanmelen]

I really like your suggestion and thanks for pointing out this edge case with the self managed node group. I forgot only EKS managed mode groups and fargate profiles automatically initialize the aws-auth configmap. This snuck by the automated tests because the complete example provisions all three node types.

What do you think about a local_exec = true and give the users the option to run kubectl locally? That will come with its own set of problems since local-exec won't support remote operations where kubectl is not install.

Another option could be using the terraform-provider-kubectl. It is not official provider but it does have a fair amount of support from the community. This might even be the more a robust solution. This approach would have the benefit of running in remote operations and it wouldn't depend on the cluster to automatically initialize the aws-auth.

[bohdantverdyi]

Would be enough , if you will add option to create only config map ( without rbac,sa,jobs ). Coz in case of self managed groups, we need configmap only.

[aidanmelen]

Solution would be to add new option to create config-map with aws-auth before job execution .

One tricky thing to consider is that this module supports replacing or patching in-place the partially managed aws-auth config map. What would happen if we have if the user provided patch = true and init-configmap? I suppose we could add a lifecycle ignore to that resource so that it only runs on first run.

[aidanmelen]

Better idea to add option to create only config map, without jobs.

I agree. If the terraform-kuberenetes-provider supported patching on create with the kuberenetes_manifest then I would have used that instead of a job.

[aidanmelen]

Would be enough , if you will add option to create only config map ( without rbac,sa,jobs ). Coz in case of self managed groups, we need configmap only.

I am working on a new release. funny enough, I implemented this logic last night. it will create a new aws-auth configmap when it does not exist and patch when it does exist.

I am going to experiment with the terraform-kubectl-provider before I push out a release.

[aidanmelen]

@bohdantverdyi this works amazingly well. I am pleasantly surprised.

locals {
  merged_map_roles = distinct(concat(
    try(yamldecode(yamldecode(var.eks.aws_auth_configmap_yaml).data.mapRoles), []),
    var.map_roles,
  ))

  aws_auth_configmap_yaml = templatefile("${path.module}/templates/aws_auth_cm.tpl",
    {
      map_roles    = local.merged_map_roles
      map_users    = var.map_users
      map_accounts = var.map_accounts
    }
  )
}

resource "kubectl_manifest" "patch" {
  override_namespace = "kube-system"
  yaml_body          = local.aws_auth_configmap_yaml
}

Could you test this branch to see if it solves your issue? I will plan on releasing tonight if it works.

[aidanmelen]

I ran a number of tests with the kubectl_manifest. it is the most robust implementation that I have used so far. It is also really simple which I think is a plus with anything that deals with authentication.

It works with only self managed node groups. all nodes. no nodes. local run. remote run. removing and adding fields.

[bohdantverdyi]

Thanks, awesome.