9627d8f commit: Error generated by "key" argument

[HugoHehn]

Having some issues with the most recent aws.s3 commit (reference 9627d8f). When running e.g. the save_object() function with AWS_ACCESS_KEY_ID, AWS_DEFAULT_REGION and AWS_SECRET_ACCESS_KEY all set as before I get the error message:

Error in if (key == "") { : argument is of length zero

Assuming that this error is generated by the new code looking for an AWS_SESSION_TOKEN even when it is not needed? Everything works if reverting to the previous commit.

Many thanks,
H

[leeper]

Thanks for reporting this. I believe it should now be fixed. You may want to pull the most recent version of aws.signature off GitHub, as well, as it contains some related fixes.

[leeper]

Oh and do let me know if this has gotten everything working correctly again!

[ChadEfaw]

Hello Thomas,

I very much appreciate your time and dedication!

I have been following these comments for the past few hours and wanted to know if the process is fixed?

I still seem to be getting the

Error in curl_fetch_memory(uri) : Couldn't connect to server Error in curl_fetch_memory(uri) : URL using bad/illegal format or missing URL Error in role[["AWS_ACCESS_KEY_ID"]] : subscript out of bounds

error when trying to use aws.s3.

Again thanks!

Chad

[leeper]

@HugoHehn @ChadEfaw @dkomd @danielerapati @willbowditch I believe this is now fixed. Changes have been made to aws.signature, aws.ec2metadata, and aws.s3, so you will need all of them from GitHub (if you're running on EC2) or the first and last (if not running on EC2). Installation status of aws.ec2metadata should now be irrelevant when not on EC2.

I've tested this and it seems to be working. I'll wait 24 hours before pushing to CRAN in case I've introduced any new bugs.

[dkomd]

Hi Thomas,
I believe I am still getting the same error as @ChadEfaw.

Thanks,

D.

[jeparsons]

Error in role[["AWS_ACCESS_KEY_ID"]] : subscript out of bounds
<Anonymous> -> s3HTTP -> locate_credentials

I've updated aws.signature, aws.ec2metadata and aws.s3, within the last 20 minutes.

[leeper]

@dkomd @jeparsons Did you restart your R session? You shouldn't be seeing that code anymore, which makes me think you're still pulling in the older version of aws.signature.

[jeparsons]

The code was called on a freshly-built docker container [had to grab the signature and metadata packages], so there shouldn't have been any residual session info.

I've since reverted to the cran package.

[HugoHehn]

@leeper I tried the new commit but continue to experience some issues which still seems to center around the aws.signature (installed the latest commit of that package as well before trying the new aws.s3 commit).

Now the aws.s3 package fails already at installation due to aws.signature 0.3.5 not being installed, but I cannot find a 0.3.5 version on GitHub or CRAN.

Downloading GitHub repo cloudyr/aws.s3@b11a366
from URL https://api.github.com/repos/cloudyr/aws.s3/zipball/b11a366
Installing aws.s3
'/usr/lib64/R/bin/R' --no-site-file --no-environ --no-save --no-restore --quiet CMD INSTALL  \
  '/mnt/tmp/RtmpqwZyJa/devtools41fd3a6aa30b/cloudyr-aws.s3-b11a366' --library='/home/rstudio/R/x86_64-redhat-linux-gnu-library/3.3'  \
  --install-tests 

* installing *source* package ‘aws.s3’ ...
** R
** inst
** tests
** preparing package for lazy loading
Error in loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]]) : 
  namespace ‘aws.signature’ 0.3.4 is being loaded, but >= 0.3.5 is required
ERROR: lazy loading failed for package ‘aws.s3’
* removing ‘/home/rstudio/R/x86_64-redhat-linux-gnu-library/3.3/aws.s3’
* restoring previous ‘/home/rstudio/R/x86_64-redhat-linux-gnu-library/3.3/aws.s3’
Installation failed: Command failed (1)

Session info:

R version 3.3.2 (2016-10-31)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: Amazon Linux AMI 2016.09

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8   
 [6] LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] aws.signature_0.3.4      githubinstall_0.2.1.9001

loaded via a namespace (and not attached):
 [1] httr_1.2.1        R6_2.2.2          tools_3.3.2       base64enc_0.1-3   withr_1.0.2       curl_2.7          memoise_1.1.0    
 [8] data.table_1.10.4 git2r_0.18.0      jsonlite_1.5      digest_0.6.12     devtools_1.13.2  

Really appreciate all the work you have put in trying to solve all this.

Best,
H.

[leeper]

@HugoHehn Sorry about that - it was a typo in the version number. I've bumped it back to where it should be. Can you try to install again?

[willbowditch]

@leeper - latest commit has fixed it for my setup. Thanks!

[HugoHehn]

@leeper Now it works excellently. Thanks a lot and sorry about not being faster with feedback the first time around.

[leeper]

@willbowditch Great!
@HugoHehn Great! No worries at all - thanks for the help getting this figured out.

@jeparsons Working for you as well?

[rmodepalli]

@leeper

Can I use IAM Role instead of access/secret keys to access S3 in R? Due to security limitation we have to use only IAM Role.

[leeper]

@rmodepalli are you talking about on an EC2 instance? If so, install the aws.ec2metadata package and the role profile will be used automatically.

[rmodepalli]

I installed aws.ec2metadata package and when I ran the listbucket getting below error. I am using in right way?

bucketlist()
Error in do.call("rbind.data.frame", r[["Buckets"]]) :
second argument must be a list

[leeper]

Check that locate_credentials() returns values you would expect. If that's working, then that means there's a bug in bucketlist(). Possibly the same as: #182

[rmodepalli]

locate_credentials() returning the output but region showing as us-east-1 but it should be us-west-1. By using the output from locate_credentials() I ran below command by changing the region then bucketlist() is working
Sys.setenv("AWS_ACCESS_KEY_ID" = "key",
"AWS_SECRET_ACCESS_KEY" = "secret",
"AWS_DEFAULT_REGION" = "us-west-1",
"AWS_SESSION_TOKEN" = "token")

Do we need to set these env variables explicitly even though we are using EC2 instance metadata?

[leeper]

@rmodepalli region is a bit picky - I'd set the environment variable or pass an explicit region = "us-west-1"` into functions just to be safe.

[rmodepalli]

@leeper When I ran the below code still giving same error. When I set env with all above 4 parameters then only its working. In production to get the accesskey, secretkey, token is not possible.

library(aws.s3)
library(aws.ec2metadata)
bucketlist(region='us-west-1')
Error in do.call("rbind.data.frame", r[["Buckets"]]) :
second argument must be a list