aws-cli fails to acquire session token before issuing sts:AssumeRole call

[cbarbour]

AWS CLI fails while attempting to issue API calls with MFA authentication. It appears to be issuing a sts:AssumeRole API call without generating or passing an appropriate session token as part of the call.

As a result, the use of MFA authentication with a role fails.

Here's the basic config:

[user]
aws_access_key_id=ABCDEFGHIJKLMNOPQRST
aws_secret_access_key=ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890ABCD

[role]
role_arn=arn:aws:iam::123456789101:/role/UserRole
source_profile=user
mfa_serial=arn:aws:iam::123456789101:mfa/user

I invoke the role like this:

aws --profile=role s3 ls --debug

Here's the debug output as a GIST:

https://gist.github.com/cbarbour/f3785cb5111009e27d6d1517b1bc5171

I've confirmed that the MFA token works fine with the web interface. I've also confirmed that I can generate a MFA authenticated session on the command line manually, allowing me to acquire the role.

I've enabled cloudtrace, and confirmed that the failed authentication attempts do not contain the appropriate credentials.

Normally, I'd expect to see a user identity like this:

            "userIdentity": {
                "accessKeyId": "...",
                "accountId": "...",
                "arn": "arn:aws:iam::...:user/...",
                "invokedBy": "signin.amazonaws.com",
                "principalId": "...",
                "sessionContext": {
                    "attributes": {
                        "creationDate": "2016-11-09T20:36:48Z",
                        "mfaAuthenticated": "true"
                    }
                },
                "type": "IAMUser",
                "userName": "..."
            }

However, when I attempt to authenticate using a role, the resulting cloudtrail entry looks like this:

        {
            "awsRegion": "us-east-1",
            "errorCode": "AccessDenied",
            "errorMessage": "Not authorized to perform sts:AssumeRole",
            "eventID": "...",
            "eventName": "AssumeRole",
            "eventSource": "sts.amazonaws.com",
            "eventTime": "2016-11-09T21:48:21Z",
            "eventType": "AwsApiCall",
            "eventVersion": "1.05",
            "recipientAccountId": "...",
            "requestID": "...",
            "requestParameters": null,
            "responseElements": null,
            "sourceIPAddress": "...",
            "userAgent": "aws-cli/1.11.13 Python/3.5.2 Darwin/15.5.0 botocore/1.4.70",
            "userIdentity": {
                "accessKeyId": "...",
                "accountId": "...",
                "arn": "arn:aws:iam::...:user/...",
                "principalId": "...",
                "type": "IAMUser",
                "userName": "..."
            }
        }

I suspect this is a Boto bug rather than an aws-cli bug, however I felt it best to file here since I haven't attempted to reproduce the problem using botocore.

[cbarbour]

Actually... I was unable to acquire the role manually either. It seems probable that this is a botocore issue rather than a aws-cli issue.

To add to this, Terraform acquires the role fine using just the profile name.

[JordonPhillips]

Alright, so I can confirm that getting assume role can be tricky. Here's what you need:

1. A user with a policy that allows them to call AssumeRole on the role you want to assume.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::ACCOUNT-ID:role/RoleToAssume"
        }
    ]
}

2. A role with a trust policy configured to allow you to assume that role, with a condition that an mfa token is required

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ACCOUNT-ID:user/Assumer"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "Bool": {
          "aws:MultiFactorAuthPresent": "true"
        }
      }
    }
  ]
}

3. Credentials in the CLI for the user

[assume-source]
aws_access_key_id = EXAMPLE
aws_secret_access_key = EXAMPLE

4. A profile with which to assume the role

[profile assume-source]
output = json
region = us-west-2
[profile assume]
output = json
region = us-west-2
role_arn = arn:aws:iam::ACCOUNT-ID:role/RoleToAssume
source_profile = assume-source
mfa_serial = arn:aws:iam::ACCOUNT-ID:mfa/Assumer

Once all that is set up, you run some command. Let's say my role gives me S3 access, so I would do this:

> aws s3 ls --profile assume
Enter MFA code: 
2016-11-14 17:39:43 somerandombucketname

Can you confirm you have a similar setup?

[cbarbour]

Hi Jordan,

Thanks for investigating. I can confirm that this is exactly the config I used.

In my previous tests, I had placed the MFA configuration in my credentials file. I tried putting the source IAM user's credentials in the credentials file and I created a profile for the source user and role in the config file. The end result was the same:

aws s3 ls --profile=assume
Enter MFA code:

An error occurred (AccessDenied) when calling the AssumeRole operation: Not authorized to perform sts:AssumeRole

I can confirm that the IAM policies are exactly as you documented. I can also confirm that my IAM user can assume the role via the GUI using MFA, and that I can assume the role on the command line using Terraform (GO AWS SDK.) The issue is specific to boto.

As an added comment, I can acquire a MFA authenticated session token by hand. Terraform / GO AWS SDK will happily use those credentials (set using environment variables,) however the AWS CLI continues to throw a Access Denied error.

I did my best to try different config permutations, to try older versions of Boto/AWS CLI, and to review the source code for both Boto and AWS cli looking for any obvious issues.

As best I can tell, Boto simply isn't acquiring an MFA authenticated session token prior to issuing the sts:AssumeRole call.

I'm not ruling out PEBCAK, but I did cover a lot of this before filing a bug.

[JordonPhillips]

I can see in your posted debug log that the token is coming from somewhere: 'TokenCode': '070702' and it does look like you're inputting it.

Could you do aws configure list --profile user ? That will show where the credentials are being sourced from.

Also note that go does not support mfa (source). Does it work if you drop the mfa token?

[cbarbour]

Hi Jordan,

The AWS CLI client correctly prompts for a MFA token; that's where the value is coming from. However, the prompted token isn't passed to AWS. This is based on the output from cloudtrail.

Note that while the MFA token is prompted for, both current/valid and invalid codes produce the same result.

The GO SDK doesn't support the MFA config option, but it will work normally if I generate a MFA authenticated session by hand. Unfortunately, the same approach doesn't work with the CLI; even if I correctly configure my environment to use a MFA authenticated session, the call to sts:AssumeRole fails and CloudTrail reports no MFA key.

E.g. if I export AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SECURITY_TOKEN, Terraform is able to acquire a role via the GO SDK, where using the exact same environment the AWS-CLI is not able t do so.

I am testing this on an account that has minimal native access; there is zero chance Terraform works without acquiring a role, and I can confirm that it fails to acquire the role throwing an error if the environment variables are not set, or not correct.

As best I can tell, boto-core simply ignores the existing session when requesting a new role session.

[aidanjl]

Im having same problem. Is this really a bug in boto-core? How is this not impacting lots of people?

[dhollema]

@cbarbour I noticed in your initial post, it appears you have an extra '/' in the role_arn.

[user]
aws_access_key_id=ABCDEFGHIJKLMNOPQRST
aws_secret_access_key=ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890ABCD

[role]
role_arn=arn:aws:iam::123456789101:/role/UserRole
source_profile=user
mfa_serial=arn:aws:iam::123456789101:mfa/user

Original:
role_arn=arn:aws:iam::123456789101:/role/UserRole

Should read:
role_arn=arn:aws:iam::123456789101:role/UserRole

I made that change on my end and authentication is failing following the same recipe as described in this thread. Without the slash, my MFA auth is fine (again, with the same recipe described in this thread). I don't think that's proper arn syntax.

[aidanjl]

Thanks @dhollema

I reviewed my config to see if I had that same problem.
I did not have an extra slash but I did have a missing colon before account ID:

My busted config:
role_arn=arn:aws:iam:123456789101:role/UserRole

Now working:
role_arn=arn:aws:iam::123456789101:role/UserRole

[Dakta]

Kicking this issue to note that this same issue can come from a range of role_arn formatting errors, including using an account alias instead of an account ID. Otherwise, recommend it be closed.

[tb3088]

Here's a twist! The AWS Cli caches the supposedly failed results in .aws/cli/cache/ and the contents are perfectly valid and usable. I extract the seesion_token and secrets from the JSON and works like a champ. The error below despite PROMPTING me for the Token and accepting it. This smells strongly of a problem in botocore.

aws sts assume-role
--role-arn arn:aws:iam::12345:role/allow-full-access-from-other-accounts
--role-session-name other-accounts@dev-full-21164
--serial-number arn:aws:iam::98765:mfa/mpatton
--duration-seconds 43200
--output text

An error occurred (AccessDenied) when calling the AssumeRole operation: MultiFactorAuthentication failed, must provide both MFA serial number and one time pass code.

[tb3088]

I probably need to open a separate issue but Botocore is doing 2 POST operations. The first one detects that the cached creds have expired and re-prompts me for a TOKEN and via a POST gets a new set of keys and session_token.

2018-12-30 18:00:08,768 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=AssumeRole) with params: {
'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8
', 'User-Agent': 'aws-cli/1.16.63 Python/3.6.4 CYGWIN_NT-10.0/2.11.2(0.329/5/3) botocore/1.12.53'}, 'body': {'Action': 'AssumeRole',
'Version': '2011-06-15', 'SerialNumber': 'arn:aws:iam::242...854:mfa/mpatton', 'RoleArn': 'arn:aws:iam::276...355:role/allow-
full-access-from-other-accounts', 'RoleSessionName': 'botocore-session-1546210796', 'TokenCode': '090009'}, 'url': 'https://sts.amaz
onaws.com/', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x6fffd5e8ac8>, 'has_strea
ming_input': False, 'auth_type': None}}

Please note the POST request it purports to generate looks like it's missing several of the data fields seen above. I don't know if that's correct.

2018-12-30 18:00:08,769 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://sts.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'aws-cli/1.16.63 Python/3.6.4 CYGWIN_NT-10.0/2.11.2(0.329/5/3) botocore/1.12.53', 'X-Amz-Date': b'20181230T230008Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAI...ZD3Q/20181230/us-east-1/sts/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=9ad1ebdad41ce0faba29302af9b1e030e2b6f412d446b1327ce342c1c84ba5cd', 'Content-Length': '249'}>

STS returns a valid HTTP Response object, so far so good?
Then Boto takes those and tries this - I can't (yet) figure out why or what it's trying to do.

2018-12-30 18:00:09,091 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://sts.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'aws-cli/1.16.63 Python/3.6.4 CYGWIN_NT-10.0/2.11.2(0.329/5/3) botocore/1.12.53', 'X-Amz-Date': b'20181230T230009Z', 'X-Amz-Security-Token': b'FQoGZXIvYXdzEFAaDJ...i0yj5m6XhBQ==', 'Authorization': b'AWS4-HMAC-SHA256 Credential=ASIAUAT....D7QM/20181230/us-east-1/sts/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token, Signature=9dd07091d7453f0...8e2828313158e8d987a08ef', 'Content-Length': '274'}>

It is this second and seemingly gratuitous request that results in the error in the prior post.

[bharathsat]

Even after i changed my aws configuration to assume the role am getting the same problem

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the AssumeRole operation: Access denied

my configuration

[default]
aws_access_key_id = XXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
[role]
role_arn = arn:aws:iam::XXXXXXXXXXXX:role/SecurityMonkeyInstanceProfile
source_profile = default

[scalp42]

Same issue here, even without MFA set up. Running into AccessDenied errors when using -profile.

If I try to use aws sts assume-role it works fine.

[VierenV]

Same problem here, my config :

/.aws/credentials

[default]
aws_access_key_id = ACCESS_KEY_1
aws_secret_access_key = SECRET_KEY_1
[other_profile]
aws_access_key_id = ACCESS_KEY_2
aws_secret_access_key = SECRET_KEY_2

/.aws/config

[default]
region = eu-west-1
[profile other_profile_role]
region = eu-west-1
role_arn = arn:aws:iam::ACCOUNT_ID:role/ROLE_TO_ASSUME
source_profile=other_profile

other_profile IAM policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::ACCOUNT_ID:role/ROLE_TO_ASSUME"
        }
    ]
}

ROLE_TO_ASSUME IAM trust policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "eks.amazonaws.com",
        "AWS": "arn:aws:iam::ACCOUNT_ID:user/other_profile"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Then the command

aws sts assume-role --profile other_profile_role --role-arn arn:aws:iam::ACCOUNT_ID:role/ROLE_TO_ASSUME --role-session-name "rolesession1"

produces
An error occurred (AccessDenied) when calling the AssumeRole operation: Access denied

[fengyali]

can this design be worse?

[aangelisc]

I have a similar problem to above..it seems that the command aws sts assume-role fails when providing the parameters serial-number and token-code, however without these the assume role succeeds. I have MFA enabled on my account and have tested this across multiple accounts.

[tb3088]

So if '--serial-number' is omitted (aka inferred from ~/.aws/config) and the token-code is prompted for interactively, or alternatively '--profile' is not specified (inferred from env:AWS_PROFILE), the command succeeds reliably, like this?

+ aws sts --debug assume-role --role-arn=arn:aws:iam::2755:role/allow-full-access-from-other-accounts --role-session-name=allow-full-access-from-other-a@dev-full.14600 --output text

Ok, I can confirm.

Evidently there is considerable neglect to check and validate inputs within the Boto functions. For instance --duration-seconds=28800 is perfectly valid per the documentation except it's not. Only under 3600 is correct. Amusingly the first API call to get the session ignores or happily accepts any value and returns a session with an expiry of 3600. Yet the 2nd API call throws a fatal error. (Hey Amazon, kindly get your service APIs to behave the same across the family of STS calls, eh?)

I also had a case where some AWS_SESSION_* variables were empty or malformed while I was testing. The Boto code didn't bother to check any of them for validity (many cases of the source code checks against 'None' instead of 'not blank or None') and submitted them to the API only for it to complain about not seeing the expected payload format.

BTW I finally pin-pointed what this 2nd web call was for - validating and effectively re-fetching the XML describing the session object in order to then print the contents. The logic is such that despite JUST having gotten a session object from the API (XML) and saved it to the local cache in JSON, or having loaded and validated a cached session (json), Boto makes this 2nd API call using 'X-Amz-Security-Token' to once again "describe the session" to itself.

Clearly someone didn't think thru the program flow properly.

It's during this second call that if you had specified '--serial-number' on the command-line (or I guess '--profile' too) that the previously resident values and MFA Tokens are not carried over. Why in the world a "validate/describe" of 'X-Amz-Security-Token' requires MFA at all is beyond me. The Token is either valid, invalid, or expired.

2019-05-29 10:27:16,422 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://sts.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'aws-cli/1.16.63 Python/3.6.4 CYGWIN_NT-10.0/2.11.2(0.329/5/3) botocore/1.12.53', 'X-Amz-Date': b'20190529T142716Z', 'X-Amz-Security-Token': b'FQoGxxxd/9Ci6pbrnBQ==', 'Authorization': b'AWS4-HMAC-SHA256 Credential=ASIxxxOF/20190529/us-east-1/sts/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token, Signature=b87cc9bxxx379618', 'Content-Length': '211'}>

[aangelisc]

@tb3088

If I omit serial number and token-code the command succeeds, however I do not get prompted for an MFA token (which shouldn't happen as MFA is enabled on the account). When using multiple profiles, if the profile flag is omitted then it succeeds as expected.

So if '--serial-number' is omitted (aka inferred from ~/.aws/config) and the token-code is prompted for interactively, or alternatively '--profile' is not specified (inferred from env:AWS_PROFILE), the command succeeds reliably, like this?

+ aws sts --debug assume-role --role-arn=arn:aws:iam::2755:role/allow-full-access-from-other-accounts --role-session-name=allow-full-access-from-other-a@dev-full.14600 --output text

Ok, I can confirm.

Evidently there is considerable neglect to check and validate inputs within the Boto functions. For instance --duration-seconds=28800 is perfectly valid per the documentation except it's not. Only under 3600 is correct. Amusingly the first API call to get the session ignores or happily accepts any value and returns a session with an expiry of 3600. Yet the 2nd API call throws a fatal error. (Hey Amazon, kindly get your service APIs to behave the same across the family of STS calls, eh?)

I also had a case where some AWS_SESSION_* variables were empty or malformed while I was testing. The Boto code didn't bother to check any of them for validity (many cases of the source code checks against 'None' instead of 'not blank or None') and submitted them to the API only for it to complain about not seeing the expected payload format.

BTW I finally pin-pointed what this 2nd web call was for - validating and effectively re-fetching the XML describing the session object in order to then print the contents. The logic is such that despite JUST having gotten a session object from the API (XML) and saved it to the local cache in JSON, or having loaded and validated a cached session (json), Boto makes this 2nd API call using 'X-Amz-Security-Token' to once again "describe the session" to itself.

Clearly someone didn't think thru the program flow properly.

It's during this second call that if you had specified '--serial-number' on the command-line (or I guess '--profile' too) that the previously resident values and MFA Tokens are not carried over. Why in the world a "validate/describe" of 'X-Amz-Security-Token' requires MFA at all is beyond me. The Token is either valid, invalid, or expired.

2019-05-29 10:27:16,422 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://sts.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'aws-cli/1.16.63 Python/3.6.4 CYGWIN_NT-10.0/2.11.2(0.329/5/3) botocore/1.12.53', 'X-Amz-Date': b'20190529T142716Z', 'X-Amz-Security-Token': b'FQoGxxxd/9Ci6pbrnBQ==', 'Authorization': b'AWS4-HMAC-SHA256 Credential=ASIxxxOF/20190529/us-east-1/sts/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token, Signature=b87cc9bxxx379618', 'Content-Length': '211'}>

[tb3088]

the AWS CLI doesn't query your account to see if it needs to prompt you for an MFA. It is simply keying off of whether 'mfa_serial' is present in the AWS_CONFIG_FILE and AWS_PROFILE stanza you're using.

[ghost]

In my instance, a subtle casing issue was the culprit, after following a template issued by a system administrator and. Note that the MFA device ARN is case sensitive. Copy and paste into your config file, to be on the safe side.

You can confirm your MFA device is working on cli with:

aws sts get-session-token --serial-number [Assigned MFA device ARN] --token-code=[Current displaying token]

Might give you some faith (as it did me).

[gadelkareem]

An error occurred (AccessDenied) when calling the AssumeRole operation: MultiFactorAuthentication failed with invalid MFA one time pass code.

I am also experiencing this error while running aws sts assume-role in a loop to get several tokens. I fixed it by adding a retry bash function but it would be better to find a real fix.

[zhujian0805]

looks like need first to get the temporary credentials for the source profile, then use those credentials to assume to target profile/account.

# First retreive the tokens for source profile with MFA
aws sts get-session-token --profile source_profile --serial-number arn:aws-cn:iam::NNNNNNNNNN:mfa/user --token-code 258731

# Or use default profile
aws --profile default sts get-session-token --serial-number arn:aws-cn:iam::NNNNNNNNNN:mfa/user --token-code 258731

# Export credentials from last step
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export AWS_SESSION_TOKEN=

# Then assume to target role
aws sts assume-role --role-arn arn:aws-cn:iam::NNNNNNNNNNNNN:role/assume-role-name --role-session-name aws-long-assume --output json

# then export credeantials from last step
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export AWS_SESSION_TOKEN=

# then you can do normal operation on the target profile/account

[kdaily]

Looks like this has been addressed with a number of suggestions.

@cbarbour @gadelkareem are you still having trouble?

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.

[terem42]

I have this issue, intermittent "MultiFactorAuthentication failed with invalid MFA one time pass code. " error.
Solved with repeat over exponential backoff delay.