Unable to use AWS ECS task role credentials

[NickLavrov]

IAM provider source code:

minio-py/minio/credentials/providers.py

Line 187 in 1ee647e

class IAMProvider(Provider):

The issue is that creds_path = "/latest/meta-data/iam/security-credentials" is hardcoded.

On an EC2 instance, you can run curl http://169.254.169.2data/iam/security-credentials to get the iam role, then curl http://169.254.169.2data/iam/security-credentials/ROLENAME to get the credentials.

For an ECS task running with a task iam role on an ec2 instance, you can get credentials if you run curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI. The value for AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is typically like: /v2/credentials/54da6c7a-84ff-41dc-a160-21b8a8bc7fb0. The provider lets you set endpoint, but since creds_path is hardcoded, I cannot get the credentials.

Also, the logic for "Get credentials of first role." is not required in the ECS task method.

I suspect a fix would involve either making creds_path also configurable (and also the logic to get the first role, since it directly gives the credentials), or checking if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is present in the environment variables and changing behavior accordingly.

Example results from curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI:

{"RoleArn":"arn:aws:iam::ACCOUNTID:role/ROLENAME","AccessKeyId":"ASIAXXXX","SecretAccessKey":"XXXXXX","Token":"TOKENTEXT","Expiration":"2020-08-13T06:11:55Z"}

[balamurugana]

@NickLavrov

For an ECS task running with a task iam role on an ec2 instance, you can get credentials if you run curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI. The value for AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is typically like: /v2/credentials/54da6c7a-84ff-41dc-a160-21b8a8bc7fb0. The provider lets you set endpoint, but since creds_path is hardcoded, I cannot get the credentials.

Is the value ``/v2/credentials/54da6c7a-84ff-41dc-a160-21b8a8bc7fb0fixed for any EC2 instance? If not, we can takeAWS_CONTAINER_CREDENTIALS_RELATIVE_URI` path as optional argument.

Also, the logic for "Get credentials of first role." is not required in the ECS task method.

It looks like ECS role is different than IAM. In that case, we may need to have different provider like IAM provider.

[NickLavrov]

@balamurugana I was just giving an example value for AWS_CONTAINER_CREDENTIALS_RELATIVE_URI.

I searched this org, and this is already implemented in golang: https://github.com/minio/minio-go/blob/f0e2f3ae36786e5e69f34fdaff106223ddba6082/pkg/credentials/iam_aws.go#L113

Also ECS task role is treated the same as IAM instance roles. Pretty much every single AWS SDK will automatically use it as a default if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is present. See the PR that added it to the minio-go minio/minio-go#1185

So that needs to be ported over to python. I can take a stab at it this week.

[NickLavrov]

The more I look at it, the more this seems like it can become a breaking change if I try to keep with the same IAMProvider. Even though most aws sdks would treat looking up credentials the same, since IAMProvider takes the endpoint argument as just the base url, and not the full path to the credentials, there will be an issue unless I add another argument to this provider:

Right now, the code takes endpoint and appends "/latest/meta-data/iam/security-credentials" as a hardcoded value to it (let's call this url1). It then expects to receive a list of IAM roles, and it takes the first one, then appends that name to url1 to get a final url from which it can obtain IAM credentials.

For ECS, this step is not needed: i.e. you would just use the url like endpoint+creds_path, where creds_path is the value of os.environ.get("AWS_CONTAINER_CREDENTIALS_RELATIVE_URI").

I can test os.environ.get("AWS_CONTAINER_CREDENTIALS_RELATIVE_URI") is not None to determine if I should use the ECS method or the EC2 method.

If a user has not defined an endpoint, then I can make these changes just fine. However, if they have defined an endpoint, AND if they are running their code in an ECS task, then they probably have their code working with the EC2 url procedure. My change would see it's in ECS and try to use the ECS url procedure, which is something they probably wouldn't have.

I can get around this 2 ways:

1. Add another argument to IAMProvider init method. It could be a Boolean is_ecs that defaults to False. This would keep functionality and allow ECS users to fetch credentials.
2. Add another provider like IAMEcsProvider. This is a bit clunky though, as I see the name for IAMProvider used to be IamEc2MetaData per https://github.com/minio/minio-py/pull/901/files. However, using a Chain should ensure this can work just fine.

Do you think 1 or 2 (or some other way) is better? From your last comment it seems you could support either the optional argument or the new provider.