Guidelines for IAM/credentials for accessing S3?

[camrinbraun]

Is there some documentation somewhere that shows exactly how to setup and use IAM roles in order to get permissions for reading/writing S3? I can manually set the credentials via

Sys.setenv("AWS_ACCESS_KEY_ID" = creds[[1]],
           "AWS_SECRET_ACCESS_KEY" = creds[[2]],
           "AWS_DEFAULT_REGION" = creds[[3]])

Then I can use aws.s3 functionality just fine. But it's unclear to me how aws.s3 looks for my credentials automatically and would get them from the IAM role I've launched with. Do I need to specify the IAM role somehow from aws.ec2metadata before even loading aws.s3?

[leeper]

You're running an EC2 instance? If you have aws.ec2metadata installed (you don't need to load it explicitly), then aws.signature::locate_credentials() will recognize that you're running on an instance and find the credentials. If you set credentials explicitly - as you've done there, or through function arguments - those will override the instance metadata. If you don't set them, the credentials in the instance metadata will be used.

[camrinbraun]

Yes running an EC2 instance. So I just tried launching a new instance with my IAM role and when I run aws.signature::locate_credentials() I just get

$key
NULL

$secret
NULL

$session_token
NULL

$region
[1] "us-east-1"

[leeper]

Do you aws.ec2metadata installed?

[camrinbraun]

Ok, so I messed up the install of aws.ec2metadata, but I have it now and the list output from locate_credentials() is filled. However, the region variable is "us-east-1" and that's not the region my instance is in.

[leeper]

Okay, that sounds like a bug. I'll investigate. In the meantime you can set an environment variable to override, or set region in every function call.

[leeper]

@camrinbraun Can you try reinstalling aws.signature (from GitHub) using ghit::install_github("cloudyr/aws.signature") ? This should be working correctly in development version 0.3.6. Let me know if not.

[camrinbraun]

With the new installation of aws.signature as you mentioned, locate_credentials() now recognizes my default region but doesn't seem to actually set the region to the recognized default. See the output of aws.signature::locate_credentials() below:

> aws.signature::locate_credentials()
Setting environment variable AWS_DEFAULT_REGION to 'us-west-2'
$key
[1] "key-was-here"

$secret
[1] "secret-was-here"

$session_token
[1] "token-was-here"

$region
[1] "us-east-1"

[leeper]

Thanks. Can you also update aws.ec2metadata - ghit::install_github("cloudyr/aws.ec2metadata") - and try again?

[camrinbraun]

That did it with the fresh installs of both aws.ec2metadata and aws.signature. Then credentials from IAM role can be found by aws.signature::locate_credentials(). Problem solved.

[camrinbraun]

and Thank you!!!

[SheldonTsen]

Hi @leeper ,

I am basically trying to do the same thing as @camrinbraun, where previously I was using an access key and secret key, but now I am trying to move towards roles. I already have an IAM role assigned to the EC2 instance, and I know the role is there because the following works:

list <- system("aws s3 ls s3://bucket-name/")
print(list)

I am able to see the following when I run:

creds <- aws.signature::locate_credentials(default_region = "ap-southeast-1a")
> creds
$key
[1] "key-was-here"

$secret
[1] "secret-key-was-here-too"

$session_token
[1] "" 

$region
[1] "ap-southeast-1a"

Session token is empty, and region would be "us-east-1" if I do not explicitly state otherwise.

But when I try to access the bucket using the following command:

files <- aws.s3::get_bucket_df(
  bucket = 'bucket'
)

It doesn't work. I tried running get_bucket_df after this:

Sys.setenv("AWS_ACCESS_KEY_ID" = creds[[1]],
           "AWS_SECRET_ACCESS_KEY" = creds[[2]])

I know the role has been assigned because the following returns the name of the IAM role:

iam_info <- metadata$iam_info()
> iam_info
$Code
[1] "Success"

$LastUpdated
[1] "2018-01-11T15:00:22Z"

$InstanceProfileArn
[1] "arn:aws:iam::<numbers>:instance-profile/<name-of-role>"

$InstanceProfileId
[1] "<bunch-of-alphanumerics>"

The packages were installed sometime in November 2017.

Would appreciate the help in getting this up and running! Am I missing some argument for something? Still new to all this AWS stuff!

Many thanks!

[leeper]

Can you let me know if this is still happening with the latest version of aws.ec2metadata (0.1.5)?

[sahitya27]

Hi @leeper ,

I'm using the aws.ec2metadata(0.1.5) and still seeing the same issue that @SheldonTsen reported. I'm doing the same kind of list that he is trying to do. Any help?

sessionInfo()
R version 3.4.3 (2017-11-30)
Platform: x86_64-conda_cos6-linux-gnu (64-bit)
Running under: Amazon Linux AMI 2018.03

Matrix products: default
BLAS: /opt/anaconda3/lib/R/lib/libRblas.so
LAPACK: /opt/anaconda3/lib/R/lib/libRlapack.so

locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] aws.signature_0.4.4 aws.ec2metadata_0.1.5 aws.s3_0.3.12

loaded via a namespace (and not attached):
[1] httr_1.3.1 compiler_3.4.3 R6_2.2.2 tools_3.4.3
[5] base64enc_0.1-3 curl_3.1 Rcpp_0.12.14 xml2_1.1.1
[9] jsonlite_1.5 digest_0.6.13

Thanks!

[sudhir05]

aws.signature::locate_credentials() didn't work for me, Rather using aws.ec2metadata how can we assume iam role to access the s3 bucket?
Code : chr "SignatureDoesNotMatch"
$ Message: chr "Credential should be scoped to a valid region, not 'ap-southeast-2'. "
$ Type : chr "Sender"

[jacobcvt12]

Same question - I want to assume the IAM role to access the S3 bucket