Using assume role with ecs-cli version 1.0.0

[mattmccarty]

In AWS/ECS our development/testing and production environments reside in the same region. In order to update any AWS/ECS resources in the development/testing "account", we have to login with our normal credentials and assume the role "DevelopmentTeam". The same goes for production, though the assumed role is "ProductionTeam".

Recently, we decided to upgrade to ecs-cli version 1.0.0. I noticed that I cannot use the --aws-profile parameter documented here to assume a role. When attempting to upload a new task, it gets uploaded, but not under the assumed role in the config file ~/.aws/config.

Here are my current aws and ecs-cli configuration files:

Versions

aws-cli/1.11.184 Python/2.7.13 Linux/4.13.9-200.fc26.x86_64 botocore/1.7.42
ecs-cli version 1.0.0 (bda91d5)

~/.aws/credentials

[default]
aws_access_key_id     = XXXXXXXXXXXXXXXXXXXX
aws_secret_access_key = SXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

[production]
aws_access_key_id     = XXXXXXXXXXXXXXXXXXXX
aws_secret_access_key = SXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

~/.aws/config

[default]
role_arn       = arn:aws:iam::XXXXXXXXXXXX:role/DevelopmentTeam
source_profile = default
region         = us-east-1
output         = json

[profile production]
role_arn       = arn:aws:iam::ZZZZZZZZZZZZ:role/ProductionTeam
source_profile = production
region         = us-east-1
output         = json

~/.ecs/config

version: v1
default: dev-application
clusters:
  dev-application:
    cluster: dev-application
    region: us-east-1
  prod-application:
    cluster: prod-application
    region: us-east-1

Example Command

ecs-cli compose -p dev-application -f /path/to/compose/file -c dev-application --cluster-config dev-application --aws-profile default create

Is this a bug, or are we doing this wrong?

[yegeniy]

I'm seeing a similar issue in ecs-cli 1.0.0

$ ecs-cli compose --aws-profile foo service up
SharedCredsAccessKey: shared credentials foo in ~/.aws/credentials did not contain aws_access_key_id
caused by: error when getting key of section 'foo': key 'aws_access_key_id' not exists"

and the ~/.aws/credentials file looks like this:

[default]
aws_access_key_id = ...
aws_secret_access_key = ...

[foo]
role_arn = arn:aws:iam::1234567890:role/my_TeamRole
source_profile = default

[PettitWesley]

@mattmccarty @yegeniy, please see the example that we have for assuming a role using an AWS Profile in the code here. We have a test case which tests this here.

A profile following this format would be added to your ~/.aws/credentials.

[assumeRoleWithCreds]
role_arn = assumeRoleWithCredsRoleArn
source_profile = assumeRoleWithCreds
external_id = 1234
role_session_name = assumeRoleWithCredsSessionName
aws_access_key_id = assumeRoleWithCredsAKID
aws_secret_access_key = assumeRoleWithCredsSKID

Let us know if using this works! Also, apologies for the obscurity of this example, we will add a link to it in our README so that it is easier to find.

[mattmccarty]

@PettitWesley I have updated my ~/.aws/credentials file as you requested.

[default]
role_arn              = arn:aws:iam::XXXXXXXXXXXX:role/DevelopmentTeam
source_profile        = default
role_session_name     = someSessionNameString
external_id           = someRandomString
aws_access_key_id     = XXXXXXXXXXXXXXXXXXXX
aws_secret_access_key = SXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

[production]
role_arn              = arn:aws:iam::ZZZZZZZZZZZZ:role/ProductionTeam
source_profile        = production
role_session_name     = someSessionNameStringForProduction
external_id           = someRandomStringForProduction
aws_access_key_id     = XXXXXXXXXXXXXXXXXXXX
aws_secret_access_key = SXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

The ecs-cli will still not assume the role specified in role_arn.

Command

ecs-cli ps --cluster dev-application --aws-profile default

Result

FATA[0000] Error executing 'ps Cluster 'dev-application' is not active. Ensure that it exists

[PettitWesley]

@mattmccarty I'm investigating; I'm trying to reproduce your issue- I will get back to you as soon as possible. In the mean time, if assume role doesn't work for you in v1.0, the method I described in this comment should still work and you can use it as a work around if necessary: #284 (comment). Since you're not using MFA, you can skip step 1.

Also, I want to confirm one thing- assume role used to work for you until you upgraded to v1.0 of the ECS CLI, correct?

[mattmccarty]

@PettitWesley it did work prior to version 1.0. I was using the --profile parameter instead of the --aws-profile parameter, however.

[PettitWesley]

@mattmccarty, apologies, this looks like its probably a bug. I can put up a fix on GitHub quickly, but obviously it will take some time before this patch is released in our pre-built binaries (if you want the fix as soon as possible, you'll have to build from the source code).

@mattmccarty, @yegeniy, in the mean time, here's a much better work around than I gave in my last message. If you look at config_test.go#L336, the test case that we have for assume role uses the $AWS_DEFAULT_PROFILE Env Var. Looking through config_v1.go, it looks like the bug is that we are only enabling SharedConfigState on the AWS Session object in the case where that Env Var is found.

That means that assume role will only work if you specify AWS Profile name using $AWS_DEFAULT_PROFILE; whereas in all other cases it will simply pull the Credentials from the the profile, but will ignore the assume role information.

I'll update in this issue once I've confirmed all of this, and submitted a fix. If you discover anything new, please post in this issue, to allow others to benefit as well.

Also, the following information on the order of precedence will be useful (this is from #370):

1. ECS CLI Profile Flags
   a) ECS Profile (--ecs-profile)
   b) AWS Profile (--aws-profile)
2. Environment Variables - attempts to fetch the credentials from environment variables:
   a) ECS_PROFILE
   b) AWS_PROFILE
   c) AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, Optional: AWS_SESSION_TOKEN
3. ECS Config - attempts to fetch the credentials from the default ECS Profile
4. Default AWS Profile - attempts to use credentials (aws_access_key_id, aws_secret_access_key) or assume_role (role_arn, source_profile) from AWS profile name
   a) AWS_DEFAULT_PROFILE environment variable (defaults to 'default')
5. EC2 Instance role

As you can see, $AWS_DEFAULT_PROFILE has lower precedence than the default ECS Profile.

Edit: I've confirmed that this is a bug, and that the work around to use $AWS_DEFAULT_PROFILE will work. A fix for this will go out in the next ECS CLI release.

[yegeniy]

Thank you for the details Wesley. This makes sense now.

…

On Mon, Nov 13, 2017, 14:49 Wesley Pettit ***@***.***> wrote: @mattmccarty <https://github.com/mattmccarty>, apologies, this looks like its probably a bug. I can put up a fix on GitHub quickly, but obviously it will take some time before this patch is released in our pre-built binaries (if you want the fix as soon as possible, you'll have to build from the source code). @mattmccarty <https://github.com/mattmccarty>, @yegeniy <https://github.com/yegeniy>, in the mean time, here's a much better work around than I gave in my last message. If you look at config_test.go#L336 <https://github.com/aws/amazon-ecs-cli/blob/master/ecs-cli/modules/config/config_test.go#L336>, the test case that we have for assume role uses the $AWS_DEFAULT_PROFILE Env Var. Looking through config_v1.go <https://github.com/aws/amazon-ecs-cli/blob/master/ecs-cli/modules/config/config_v1.go#L161>, it looks like the bug is that we are only enabling SharedConfigState on the AWS Session object in the case where that Env Var is found. That means that assume role will only work if you specify AWS Profile name using $AWS_DEFAULT_PROFILE; whereas in all other cases it will simply pull the Credentials from the the profile, but will ignore the assume role information. I'll update in this issue once I've confirmed all of this, and submitted a fix. If you discover anything new, please post in this issue, to allow others to benefit as well. Also, the following information on the order of precedence will be useful (this is from #370 <#370>): 1. ECS CLI Profile Flags a) ECS Profile (--ecs-profile) b) AWS Profile (--aws-profile) 2. Environment Variables - attempts to fetch the credentials from environment variables: a) ECS_PROFILE b) AWS_PROFILE c) AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, Optional: AWS_SESSION_TOKEN 3. ECS Config - attempts to fetch the credentials from the default ECS Profile 4. Default AWS Profile - attempts to use credentials (aws_access_key_id, aws_secret_access_key) or assume_role (role_arn, source_profile) from AWS profile name a) AWS_DEFAULT_PROFILE environment variable (defaults to 'default') 5. EC2 Instance role As you can see, $AWS_DEFAULT_PROFILE has the lower precedence than the default ECS Profile. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#369 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAR3wpP9-bat_3-2swE_EYUDWQic8SeUks5s2J1NgaJpZM4QYU1Q> .

[mattmccarty]

Thanks for the info @PettitWesley !

[yegeniy]

This workaround worked for me. Note that, due order of precedence, you must only have the AWS_DEFAULT_PROFILE parameter set. Not the AWS_PROFILE one

[dimisjim]

@PettitWesley @yegeniy Have tried both:
AWS_DEFAULT_PROFILE=AW_PROD_PHP ecs-cli ps --cluster prodcution --region eu-west-1
ecs-cli ps --cluster prodcution --aws-profile AW_PROD_PHP --region eu-west-1

but getting:

Error executing 'ps': AssumeRoleTokenProviderNotSetError: assume role with MFA enabled, but AssumeRoleTokenProvider session option not set.

[yegeniy]

Hey @dimisjim, I haven't really used this tool for about a year, so I don't think I can help you debug it at this time.

I recommend eliminating the possibility that you are not providing your credentials incorrectly first & ensure you can do whatever the ecs-cli ps command does without using ecs-cli to start, while using that AW_PROD_PHP user.

• I don't believe that I was trying to assume role with MFA enabled, but maybe I was... Is the presence of AssumeRoleTokenProviderNotSetError telling you that you are doing something different from what you were expecting to be doing?

It looks like this project is on version 1.13.0 by this point, though, so after debugging your approach as far as you can / have time for, you may want to consider opening a new issue and linking to this one if no one responds in a timely manner.

[PettitWesley]

@dimisjim I think you're getting that error because the CLI doesn't support prompting for an MFA token (we need to add this).

Right now, the best way to use MFA with the CLI is to call assume-role with the AWS CLI, and then configure an ECS CLI Profile with the resulting access key, secret key, and session token:

ecs-cli configure profile --profile-name profile_name --access-key <value> --secret-key <value> --session-token <value>

If anyone wants to try to add support for this in the ECS CLI, I believe that this SDK Package could be used to add this functionality. Hopefully we'll be able to work on this eventually.

[dimisjim]

@PettitWesley Thanks for the reply. Hopefully this is gonna get added soon.

[ux-engineer]

Getting same error than @dimisjim.

[ux-engineer]

Cannot even use this approach for getting session token: aws sts get-session-token --profile myprofilename

An error occurred (AccessDenied) when calling the GetSessionToken operation: Cannot call GetSessionToken with session credentials

[PettitWesley]

@envision If your profile is already based on temporary credentials, you can not call GetSessionToken again: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#stsapi_comparison