Spaces fixes #165

amoeba · 2018-08-04T06:55:19Z

Kinda-grabbag-of-a-PR here:

aws.s3 got some fixes, not sure if they're on CRAN yet, but now things that broke now work!

space_create works and is now re-enabled
spaces_object_put works w/ multipart
spaces_object_delete works now and is now re-enabled

Also added:

All regions can now be used via the DO_SPACES_REGION env var, to address Cannot change "spaces_base" variable, spaces different region support #163
Appended a note to vignette about Spaces/Objects. I see your issues about having R help pages in Make manual files to summarise droplets and spaces #150 which I agree is an improvement
Added space_delete and space_list() which are two API calls we hadn't done

Fixes #163

More to come as I make time!

@thosjleeper

@thosjleeper added this fix for us in aws.s3 so this works now

space_list returns a data.frame of the contents and space_delete does what it says on the tin

This works now!

amoeba · 2018-08-05T07:26:37Z

Just added space_location and a small fix I missed from R CMD CHECK.

amoeba · 2018-08-05T09:30:12Z

Add documentation page at ?spaces_info. Is that the kind of thing you were looking for in #150?

sckott · 2018-08-05T15:32:23Z

Looking good.

Add documentation page at ?spaces_info. Is that the kind of thing you were looking for in #150?

Yeah i think so, looks good. We can see what users are confused by and add more details later.

Are there other things you're adding here?

amoeba · 2018-08-05T20:13:54Z

All good now! Thanks.

sckott · 2018-08-06T23:47:40Z

@amoeba i can't get space_create() to work. I pulled your branch down locally, then devtools::load_all(), I had a access key and secret key and then set the region to nyc3, then did

space_create("new_space_name", config=httr::verbose())

-> PUT /new_space_name/ HTTP/1.1
-> Host: nyc3.digitaloceanspaces.com
-> User-Agent: libcurl/7.54.0 r-curl/3.2 httr/1.3.1
-> Accept-Encoding: gzip, deflate
-> Accept: application/json, text/xml, application/xml, */*
-> Content-Length: 0
->
<- HTTP/1.1 400 Bad Request
<- Date: Mon, 06 Aug 2018 23:46:23 GMT
<- Content-Length: 153
<- Content-Type: text/plain; charset=utf-8
<-

with some headers stripped out for brevity.

so i'm getting a 400 client error. Does it work for you?

amoeba · 2018-08-07T00:06:37Z

Did you build a copy of aws.s3 from the latest on the master branch or are you using the latest CRAN release?

sckott · 2018-08-07T01:37:21Z

used latest from github 0.3.19

amoeba · 2018-08-07T02:35:48Z

Okay, thanks for testing this out. I'll take a look in the AM and see if I was just hallucinating.

sckott · 2018-08-07T02:37:33Z

thanks

amoeba · 2018-08-07T02:52:34Z

I had a sec just now to test. Works for me. Our implementation doesn't catch any errors aws.s3 gets into so it's a bit of a pain and could use some work. I wonder what's different between our approaches. Here's mine:

> library(analogsea)
> Sys.setenv("DO_SPACES_REGION"="nyc3",
+            "DO_SPACES_ACCESS_KEY"="...",
+            "DO_SPACES_SECRET_KEY"="...")
> # List my spaces
> spaces()
named list()
> space_create("test-for-scott")
New space test-for-scott created successfully

Session Info

> devtools::session_info()
Session info -----------------------------------------------------------------------------
 setting  value                       
 version  R version 3.5.1 (2018-07-02)
 system   x86_64, darwin15.6.0        
 ui       RStudio (1.2.723)           
 language (EN)                        
 collate  en_US.UTF-8                 
 tz       America/Juneau              
 date     2018-08-06                  

Packages ---------------------------------------------------------------------------------
 package       * version    date       source                         
 analogsea     * 0.6.5.9110 2018-08-07 local                          
 aws.s3          0.3.19     2018-08-07 Github (cloudyr/aws.s3@874f6a7)
 aws.signature   0.4.4      2018-07-27 CRAN (R 3.5.0)                 
 base          * 3.5.1      2018-07-05 local                          
 base64enc       0.1-3      2015-07-28 CRAN (R 3.5.0)                 
 compiler        3.5.1      2018-07-05 local                          
 curl            3.2        2018-03-28 CRAN (R 3.5.0)                 
 datasets      * 3.5.1      2018-07-05 local                          
 devtools        1.13.5     2018-02-18 CRAN (R 3.5.0)                 
 digest          0.6.15     2018-01-28 CRAN (R 3.5.0)                 
 graphics      * 3.5.1      2018-07-05 local                          
 grDevices     * 3.5.1      2018-07-05 local                          
 httr            1.3.1      2017-08-20 CRAN (R 3.5.0)                 
 jsonlite        1.5        2017-06-01 CRAN (R 3.5.0)                 
 magrittr        1.5        2014-11-22 CRAN (R 3.5.0)                 
 memoise         1.1.0      2017-04-21 CRAN (R 3.5.0)                 
 methods       * 3.5.1      2018-07-05 local                          
 mime            0.5        2016-07-07 CRAN (R 3.5.0)                 
 R6              2.2.2      2017-06-17 CRAN (R 3.5.0)                 
 Rcpp            0.12.18    2018-07-23 CRAN (R 3.5.0)                 
 stats         * 3.5.1      2018-07-05 local                          
 tools           3.5.1      2018-07-05 local                          
 utils         * 3.5.1      2018-07-05 local                          
 withr           2.1.2      2018-03-15 CRAN (R 3.5.0)                 
 xml2            1.2.0      2018-01-24 CRAN (R 3.5.0)                 
 yaml            2.2.0      2018-07-25 cran (@2.2.0)
 collate  en_US.UTF-8                 
 tz       America/Juneau              
 date     2018-08-06                  

Packages ---------------------------------------------------------------------------------
 package       * version    date       source        
 analogsea     * 0.6.5.9110 2018-08-07 local         
 aws.s3          0.3.19     2018-08-07 local         
 aws.signature   0.4.1      2018-04-09 CRAN (R 3.5.0)
 base          * 3.5.1      2018-07-05 local         
 base64enc       0.1-3      2015-07-28 CRAN (R 3.5.0)
 compiler        3.5.1      2018-07-05 local         
 curl            3.2        2018-03-28 CRAN (R 3.5.0)
 datasets      * 3.5.1      2018-07-05 local         
 devtools        1.13.5     2018-02-18 CRAN (R 3.5.0)
 digest          0.6.15     2018-01-28 CRAN (R 3.5.0)
 graphics      * 3.5.1      2018-07-05 local         
 grDevices     * 3.5.1      2018-07-05 local         
 httr            1.3.1      2017-08-20 CRAN (R 3.5.0)
 jsonlite        1.5        2017-06-01 CRAN (R 3.5.0)
 magrittr        1.5        2014-11-22 CRAN (R 3.5.0)
 memoise         1.1.0      2017-04-21 CRAN (R 3.5.0)
 methods       * 3.5.1      2018-07-05 local         
 mime            0.5        2016-07-07 CRAN (R 3.5.0)
 R6              2.2.2      2017-06-17 CRAN (R 3.5.0)
 Rcpp            0.12.17    2018-05-18 CRAN (R 3.5.0)
 stats         * 3.5.1      2018-07-05 local         
 tools           3.5.1      2018-07-05 local         
 utils         * 3.5.1      2018-07-05 local         
 withr           2.1.2      2018-03-15 CRAN (R 3.5.0)
 xml2            1.2.0      2018-01-24 CRAN (R 3.5.0)
 yaml            2.2.0      2018-07-25 cran (@2.2.0)

And the space I made shows up on the website just fine. Our our versions of aws.s3 and aws.signature the same?

sckott · 2018-08-07T18:38:46Z

will check, thanks

sckott · 2018-08-07T18:54:42Z

same versions

Packages -------------
 package       * version    date       source
 .py             0.0.16     <NA>       local
 analogsea     * 0.6.5.9110 <NA>       local
 aws.s3          0.3.19     2018-08-06 Github (cloudyr/aws.s3@874f6a7)
 aws.signature   0.4.4      2018-07-27 CRAN (R 3.5.1)
 base          * 3.5.1      2018-07-19 local
 base64enc       0.1-3      2015-07-28 CRAN (R 3.5.0)
 commonmark      1.5        2018-04-28 CRAN (R 3.5.0)
 compiler        3.5.1      2018-07-19 local
 curl            3.2        2018-03-28 CRAN (R 3.5.0)
 datasets      * 3.5.1      2018-07-19 local
 devtools        1.13.6     2018-06-27 CRAN (R 3.5.0)
 digest          0.6.15     2018-01-28 CRAN (R 3.5.0)
 graphics      * 3.5.1      2018-07-19 local
 grDevices     * 3.5.1      2018-07-19 local
 httr            1.3.1      2017-08-20 CRAN (R 3.5.0)
 jsonlite        1.5        2017-06-01 CRAN (R 3.5.0)
 magrittr        1.5        2014-11-22 CRAN (R 3.5.0)
 memoise         1.1.0      2017-04-21 CRAN (R 3.5.0)
 methods       * 3.5.1      2018-07-19 local
 R6              2.2.2      2017-06-17 CRAN (R 3.5.0)
 Rcpp            0.12.18    2018-07-23 CRAN (R 3.5.1)
 rlang           0.2.1      2018-05-30 CRAN (R 3.5.0)
 roxygen2        6.1.0      2018-07-27 CRAN (R 3.5.0)
 rstudioapi      0.7        2017-09-07 CRAN (R 3.5.0)
 rtichoke        0.2.1      <NA>       local
 ssh             0.2        2018-06-04 CRAN (R 3.5.0)
 stats         * 3.5.1      2018-07-19 local
 stringi         1.2.4      2018-07-20 CRAN (R 3.5.0)
 stringr         1.3.1      2018-05-10 CRAN (R 3.5.0)
 testthat      * 2.0.0      2017-12-13 CRAN (R 3.5.0)
 tools           3.5.1      2018-07-19 local
 utils         * 3.5.1      2018-07-19 local
 withr           2.1.2      2018-03-15 CRAN (R 3.5.0)
 xml2            1.2.0      2018-01-24 CRAN (R 3.5.0)
 yaml            2.2.0      2018-07-25 CRAN (R 3.5.1)

Still not workfing for me, can you do space_create() with config=httr::verbose() so I can see the curl details? I'm getting a 400 but because aws.s3 just returns NULL on an error and doesn't pass on the response body I can't see what the message is, a downside of depending on this external pkg

amoeba · 2018-08-07T20:16:32Z

Here's the doubly-verbose output:

> space_create("test-for-scott", config=httr::verbose(), verbose = TRUE)
Checking for credentials in user-supplied values
Using user-supplied value for AWS Access Key ID
Using user-supplied value for AWS Secret Access Key
Using user-supplied value for AWS Region ('nyc3')
Non-AWS base URL requested.
S3 Request URL: https://nyc3.digitaloceanspaces.com/test-for-scott/
Executing request with AWS credentials
Checking for credentials in user-supplied values
Using user-supplied value for AWS Access Key ID
Using user-supplied value for AWS Secret Access Key
Using user-supplied value for AWS Region ('nyc3')
Checking for credentials in user-supplied values
Using user-supplied value for AWS Secret Access Key
Using user-supplied value for AWS Region ('nyc3')
-> PUT /test-for-scott/ HTTP/1.1
-> Host: nyc3.digitaloceanspaces.com
-> User-Agent: libcurl/7.54.0 r-curl/3.2 httr/1.3.1
-> Accept-Encoding: gzip, deflate
-> Accept: application/json, text/xml, application/xml, */*
-> x-amz-acl: private
-> x-amz-date: 20180807T201511Z
-> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
-> Authorization: AWS4-HMAC-SHA256 Credential=P4UB7DRO2X7VX2UV47LN/20180807/nyc3/s3/aws4_request, SignedHeaders=host;x-amz-acl;x-amz-date, Signature=85381e20229d9359ad4361b2b3929074c1cc52ea86e269d7c17af627dc772729
-> Content-Length: 0
-> 
<- HTTP/1.1 200 OK
<- Date: Tue, 07 Aug 2018 20:15:11 GMT
<- Content-Length: 0
<- 
Parsing AWS API response
Success: (200) OK
New space test-for-scott created successfully

I've found debugging by throwing browser() statements in s3HTTP which does most of the work. I'm scratching my head trying to think how I can help debug further.

sckott · 2019-03-21T00:10:10Z

hugh, seems that I was running into invalid regions

https://www.digitalocean.com/docs/platform/availability-matrix/#product-availability-by-datacenter-region suggests nyc3 is not available, with the following avail: ams3, sfo2, sgp1

i imagine we should change the default to one of those 3?

amoeba · 2019-03-21T00:18:16Z

i imagine we should change the default to one of those 3?

I think this is just due to today's spaces outage, and I think it might fix itself. IIRC nyc3 was the first and only availability region and I haven't ehard anything that indicates it's going away.

Incident details: https://status.digitalocean.com/incidents/bbyt03t1vm88

I'll do some digging.

sckott · 2019-03-21T00:19:53Z

hmm, if that's right that nyc3 is just out temporarily, then I guess I don't know why it was failing on my end before. Anyway, seems to be working now

sckott · 2019-03-21T00:21:24Z

i can work on #150 and then I think we're very close to getting this off to cran

amoeba · 2019-03-21T00:33:56Z

Sounds great, let me know if you want a hand or a second set of eyes.

amoeba added 8 commits August 3, 2018 22:42

Expand Spaces implementation so it works across regions

f0f9bfa

Fix and re-enable space creation

7c7b3c5

@thosjleeper added this fix for us in aws.s3 so this works now

Return the name of the space that's creation in space_create

214e8a1

Add space_list and space_delete

dc2b07b

space_list returns a data.frame of the contents and space_delete does what it says on the tin

Turn spaces_object_delete back on now that it works

5211021

Remove comment about spaces_object_put not working w/ multipart

53f29b6

This works now!

Redocument the package

a61ce59

Drop a note in the vignette about Spaces stuff

f990e5e

amoeba mentioned this pull request Aug 5, 2018

Cannot change "spaces_base" variable, spaces different region support #163

Closed

amoeba added 2 commits August 4, 2018 23:25

Add space_location (get space region)

7ba445d

Switch ::: to :: in call to get_bucket_df I shouldn't've had or missed

0057d14

amoeba added 2 commits August 5, 2018 00:47

Partia docs overhaul

4a7eb4f

Fix various doc issues related to previous commits

aaa30c0

sckott added this to the v0.7 milestone Aug 5, 2018

sckott added the spaces label Dec 19, 2018

amoeba force-pushed the spaces-fixes branch from 7aea4fa to aaa30c0 Compare March 21, 2019 00:32

Merge branch 'master' into spaces-fixes

81ded52

sckott merged commit 59ad079 into pachadotdev:master Mar 21, 2019

sckott modified the milestones: v0.7, v0.8 Jun 3, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Spaces fixes #165

Spaces fixes #165

amoeba commented Aug 4, 2018

amoeba commented Aug 5, 2018

amoeba commented Aug 5, 2018

sckott commented Aug 5, 2018

amoeba commented Aug 5, 2018

sckott commented Aug 6, 2018

amoeba commented Aug 7, 2018

sckott commented Aug 7, 2018

amoeba commented Aug 7, 2018

sckott commented Aug 7, 2018

amoeba commented Aug 7, 2018

sckott commented Aug 7, 2018

sckott commented Aug 7, 2018

amoeba commented Aug 7, 2018

sckott commented Mar 21, 2019

amoeba commented Mar 21, 2019

sckott commented Mar 21, 2019

sckott commented Mar 21, 2019

amoeba commented Mar 21, 2019

Spaces fixes #165

Spaces fixes #165

Conversation

amoeba commented Aug 4, 2018

amoeba commented Aug 5, 2018

amoeba commented Aug 5, 2018

sckott commented Aug 5, 2018

amoeba commented Aug 5, 2018

sckott commented Aug 6, 2018

amoeba commented Aug 7, 2018

sckott commented Aug 7, 2018

amoeba commented Aug 7, 2018

sckott commented Aug 7, 2018

amoeba commented Aug 7, 2018

sckott commented Aug 7, 2018

sckott commented Aug 7, 2018

amoeba commented Aug 7, 2018

sckott commented Mar 21, 2019

amoeba commented Mar 21, 2019

sckott commented Mar 21, 2019

sckott commented Mar 21, 2019

amoeba commented Mar 21, 2019