[EKS] [aws-auth]: Allow customer to rollback aws-auth configmap when update the configuration wrong

[zweizhang]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[zweizhang]

Tell us about your request
Provide a feature for customer to rollback or update configmap of aws-auth when creator update the configmap wrong

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
After created EKS cluster done, we need update aws-auth file to allow some users or some roles to access EKS cluster.
If the creator update the configmap of aws-auth wrong, like adding his roles to the configmap, then the cluster creator can't access the EKS cluster any more. It is terriable.

Are you currently working around this issue?
When the issue happen, we need create a aws case ticket for the backend team to correct the aws-auth setting, if will take a long time. If the creator can rollback the aws-auth file to the older version by himself, it will be really helpful.

Additional context
The creator can't manage EKS resource when update aws-auth configmap wrong.
$ kubectl get nodes
error: You must be logged in to the server (Unauthorized)

[GnatorX]

The role that created the EKS cluster will always have admin access back into the EKS cluster. You may have broken all other role's access but you should be able to go back in as that role and reapply aws-auth.(solution for now)

[zweizhang]

@GnatorX Yes, I knew that the role that created the EKS cluster will always have admin access back into the EKS cluster.
The key point is if the above role was add into the aws-auth file with a wrong setting, after appled, the admin role can't manage the EKS cluster any more. You can test it.
Because the EKS master cluster managed by AWS team, so cutomer can't rollback nor update the aws-auth file by himself/herself.
Why does the creator add the creator role into aws-auth file ? No one know it, but it was a issue for customer.

So one way to avoid such issue is that don't allow user to add the creator role into aws-auth file ?
Another way is that allow user to rollback the wrong configuration of aws-auth file by himself/herself ?

thanks in advance.

[GnatorX]

Totally agree just wanted to add it for people who ran into this and are looking for a solution.

[mikestef9]

This will be solved by #185, which is an EKS API replacement for the aws-auth config map

[Arsen-Uulu]

role

Not true

[melvinRspectrumio]

We had to call AWS support to get this fixed! which is a bummer as we can't do anything ourselves.

[m1mohamad]

I made a mistake and updated my cluster with the following configmap while I was trying to test why nodes previously arent getting READY , so I lost access to the cluster, then I used the same role below to access the cluster again but I lost my system:masters permissions ! I can't get do kubectl get configmaps , is there a way to regain access ?

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: 'arn:aws:iam::xxxxxxx:role/testyy-NodeInstanceRole-1FQVVVZPS0TDP'
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes

[mikestef9]

Addressed with #185