When using MFA enabled IAM users, eksctl is not caching MFA token and is requesting for a new token for every command execution.

[skkumsi]

What happened?

When AWS CLI is configured with MFA enabled IAM users, for every eksctl command execution a prompt for MFA token is being given out. The 'eksctl' client is not caching the token unlike AWS CLI where the MFA token credentials are cached under ~/.aws/cli/cache directory.

How to reproduce it?

file: ~/.aws/config

[profile tester]
output = json
region = us-east-1
mfa_serial = arn:aws:iam::36xxxxxxx:mfa/tester
role_arn = arn:aws:iam::36xxxxxxx:role/awsqs-eks-cluster
source_profile = tester

In the above CLI config the MFA enabled IAM user 'tester' is assuming an IAM Role 'awsqs-eks-cluster'

when 'aws sts get-caller-identity' is executed for the first time, the MFA token prompt is being requested. Upon providing a valid MFA token, the command returns the following output.

$ aws sts get-caller-identity
Enter MFA code for arn:aws:iam::36xxxxxxx:mfa/tester:
{
    "UserId": "AROAVKP2SENF2BUEXAMPLE:botocore-session-1628114328",
    "Account": "36xxxxxxx",
    "Arn": "arn:aws:sts::36xxxxxxx:assumed-role/awsqs-eks-cluster/botocore-session-1628114328"
}

$ ls -l ~/.aws/cli/cache/
total 8
-rw-------  1 admin  staff  1410 Aug  4 17:59 6b8b3ff506784382ca184e327173d67cdb3bf574.json

For all consecutive AWS CLI commands, the cached credentials are used and users needs not enter MFA token again. If the above mentioned credentials.json file is deleted, AWS CLI also prompts for a MFA token again.

But in case of 'eksctl' client for every execution MFA token prompt is being given out. The 'eksctl' client is not using the cached credentials in ~/.aws/cli/cache/ , neither storing them after first execution elsewhere.

$ eksctl get clusters --name eks-cluster-name
Assume Role MFA token code: 525923
2021-08-04 18:07:38 [i]  eksctl version 0.59.0
2021-08-04 18:07:38 [i]  using region us-east-1
NAME			VERSION	STATUS	CREATED			VPC			SUBNETS							SECURITYGROUPS
eks-cluster-name	1.18	ACTIVE	2021-01-08T13:22:03Z	vpc-0de0dd0example	subnet-050b704bexample,subnet-0a8565example

$ eksctl get clusters --name eks-cluster-name
Assume Role MFA token code: 066025
2021-08-04 18:08:06 [i]  eksctl version 0.59.0
2021-08-04 18:08:06 [i]  using region us-east-1
NAME			VERSION	STATUS	CREATED			VPC			SUBNETS							SECURITYGROUPS
eks-cluster-name	1.18	ACTIVE	2021-01-08T13:22:03Z	vpc-0de0dd0example	subnet-050b704bexample,subnet-0a8565example

$ eksctl get clusters --name eks-cluster-name
Assume Role MFA token code: 609887
2021-08-04 18:08:39 [i]  eksctl version 0.59.0
2021-08-04 18:08:39 [i]  using region us-east-1
NAME			VERSION	STATUS	CREATED			VPC			SUBNETS							SECURITYGROUPS
eks-cluster-name	1.18	ACTIVE	2021-01-08T13:22:03Z	vpc-0de0dd0example	subnet-050b704bexample,subnet-0a8565example

Are you using a downloaded binary or did you compile eksctl?
Downloaded binary

What type of AWS credentials are you using (i.e. default/named profile, MFA)? -
MFA.

Versions

$ eksctl info
eksctl version: 0.59.0
kubectl version: v1.18.6
OS: darwin

[Callisto13]

Thanks for asking @skkumsi, we will look into this.

Eksctl does not do any authentication itself, it simply delegates to whatever is happening in your local environment. So there is a chance that we can fix this with some changes to your env. We will also try to reproduce and rule out eksctl as the issue.

[binchenX]

I think the "problem" is due to ekstctl didn't do any caching of the token[1]. It would requires some effort to implement it. [2][3].

[1] aws/aws-sdk-go#1329
[2] https://github.com/wallix/awless/blob/44e892b4961fc09abf82f61de2ffeb66a20c82b7/aws/services/credentials_providers.go#L51
[3] kubernetes-sigs/aws-iam-authenticator#193

[Skarlso]

Verified that nothing I do to set in aws environment (i.e. session tokens, assumed role creds, session values) can make eksctl not ask for the mfa token over and over again.

Going to look into caching next.

[Skarlso]

Hmm, according to #57 this should already work.

[Skarlso]

Ah, no, that just fixes the AWS_PROFILE thing. Here is the relevant comment from Richard:

#57 (comment)

If using MFA this will ask for a code everytime its run and doesn't cache them between runs.

Ok, so it's apparent that caching is not going to happen. Just wanted to make sure, I'm not missing something.

And yet, the pr description is super confusing:

If endpoints are being overriden then the credentials from the initial
session creation are shared with any subsequent session creation to
ensure that the tokens are shared (otherwise you may get multiple MFA
prompts).

This would suggest that the session should be forwarded, but isn't.

[Skarlso]

Looking at the code, this actually should already work. Interesting.

[Skarlso]

Nope, I understand now what needs to be done.

[Skarlso]

@skkumsi thank you for the detailed issue links, it really helped in finding and putting this together.

[Skarlso]

Once released, please test and feel free to re-open this in any case there is a question or issue with the cache.