Print methods for http verb with the Paginator client

[boshek]

Thanks for such a great project.

Is there anyway to customize the print methods for a get http verb with a Pagination request? Right now each pagination step seems to produce a new line. As a quick and dirty progress bar, I was wondering if one could take the total number of records you are after divided by the limit_chunk to estimate where you are with respect to some data retrieval operation. Here is what I see now in a package that uses crul under the hood. Possibly silly question but I can't figure out where get print methods for the Paginator are defined to poke around here.

>   ## devtools::install_github("bcgov/bcdata")
>   
>   library(bcdata)
>   bcdc_get_geodata("sites-registry-open-government-license-")
This record request pagination to complete the request.
Retrieving data




OK

Parsing data
Simple feature collection with 10023 features and 29 fields
geometry type:  POINT
dimension:      XY
bbox:           xmin: 325008.9 ymin: 367957.6 xmax: 1799019 ymax: 1710843
epsg (SRID):    3005
proj4string:    +proj=aea +lat_1=50 +lat_2=58.5 +lat_0=45 +lon_0=-126 +x_0=1000000 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs
# A tibble: 10,023 x 30
   id    CUSTODIAN_ORG_D~ BUSINESS_CATEGO~ BUSINESS_CATEGO~ BUSINESS_CATEGO~ OCCUPANT_TYPE_ID OCCUPANT_TYPE_D~ CUSTOM_STYLE_NA~
   <chr> <chr>                       <int> <chr>            <chr>                       <int> <chr>            <chr>           
 1 WHSE~ DestinationBC                  30 accommodationSe~ Accommodation s~               13 HelloBC Accomod~ NA              
 2 WHSE~ DestinationBC                  22 artsEntertainme~ Arts, entertain~               12 HelloBC Activit~ NA              
 3 WHSE~ DestinationBC                  22 artsEntertainme~ Arts, entertain~               12 HelloBC Activit~ NA              
 4 WHSE~ DestinationBC                  22 artsEntertainme~ Arts, entertain~               12 HelloBC Activit~ NA              
 5 WHSE~ DestinationBC                  22 artsEntertainme~ Arts, entertain~               12 HelloBC Activit~ NA              
 6 WHSE~ DestinationBC                  22 artsEntertainme~ Arts, entertain~               12 HelloBC Activit~ NA              
 7 WHSE~ DestinationBC                  22 artsEntertainme~ Arts, entertain~               12 HelloBC Activit~ NA              
 8 WHSE~ DestinationBC                  22 artsEntertainme~ Arts, entertain~               12 HelloBC Activit~ NA              
 9 WHSE~ DestinationBC                  22 artsEntertainme~ Arts, entertain~               12 HelloBC Activit~ NA              
10 WHSE~ DestinationBC                  22 artsEntertainme~ Arts, entertain~               12 HelloBC Activit~ NA              
# ... with 10,013 more rows, and 22 more variables: SOURCE_DATA_ID <chr>, OCCUPANT_NAME <chr>, DESCRIPTION <chr>,
#   PHYSICAL_ADDRESS <chr>, ALIAS_ADDRESS <chr>, POSTAL_CODE <chr>, LOCALITY <chr>, CONTACT_PHONE <chr>, CONTACT_EMAIL <chr>,
#   CONTACT_FAX <chr>, WEBSITE_URL <chr>, IMAGE_URL <chr>, LATITUDE <dbl>, LONGITUDE <dbl>, KEYWORDS <chr>,
#   NON_CIVIC_ADDRESS_IND <chr>, LOCATION_DESCRIPTOR <chr>, DATE_ADDED <chr>, DATE_UPDATED <chr>, OBJECTID <int>,
#   SE_ANNO_CAD_DATA <chr>, geometry <POINT [m]>

Session Info

- Session info ----------------------------------------------------------------------------------------------------------------------
 setting  value                       
 version  R version 3.5.2 (2018-12-20)
 os       Windows 10 x64              
 system   x86_64, mingw32             
 ui       RStudio                     
 language (EN)                        
 collate  English_Canada.1252         
 ctype    English_Canada.1252         
 tz       America/Los_Angeles         
 date     2019-03-25                  

- Packages --------------------------------------------------------------------------------------------------------------------------
 package     * version    date       lib source                       
 assertthat    0.2.1      2019-03-21 [1] CRAN (R 3.5.2)               
 backports     1.1.3      2018-12-14 [1] CRAN (R 3.5.2)               
 bcdata      * 0.0.0.9000 2019-03-22 [1] Github (bcgov/bcdata@2ed5cbf)
 callr         3.2.0      2019-03-15 [1] CRAN (R 3.5.3)               
 class         7.3-15     2019-01-01 [1] CRAN (R 3.5.3)               
 classInt      0.3-1      2018-12-18 [1] CRAN (R 3.5.3)               
 cli           1.1.0      2019-03-19 [1] CRAN (R 3.5.3)               
 clipr         0.5.0      2019-01-11 [1] CRAN (R 3.5.3)               
 crayon        1.3.4      2017-09-16 [1] CRAN (R 3.5.3)               
 crul          0.7.0      2019-01-04 [1] CRAN (R 3.5.3)               
 curl          3.3        2019-01-10 [1] CRAN (R 3.5.3)               
 DBI           1.0.0      2018-05-02 [1] CRAN (R 3.5.3)               
 dbplyr        1.3.0      2019-01-09 [1] CRAN (R 3.5.3)               
 desc          1.2.0      2018-05-01 [1] CRAN (R 3.5.3)               
 devtools      2.0.1      2018-10-26 [1] CRAN (R 3.5.3)               
 digest        0.6.18     2018-10-10 [1] CRAN (R 3.5.3)               
 dplyr         0.8.0.1    2019-02-15 [1] CRAN (R 3.5.3)               
 e1071         1.7-1      2019-03-19 [1] CRAN (R 3.5.3)               
 evaluate      0.13       2019-02-12 [1] CRAN (R 3.5.3)               
 fansi         0.4.0      2018-10-05 [1] CRAN (R 3.5.3)               
 fs            1.2.7      2019-03-19 [1] CRAN (R 3.5.3)               
 glue          1.3.1      2019-03-12 [1] CRAN (R 3.5.3)               
 htmltools     0.3.6      2017-04-28 [1] CRAN (R 3.5.3)               
 httpcode      0.2.0      2016-11-14 [1] CRAN (R 3.5.2)               
 httr          1.4.0      2018-12-11 [1] CRAN (R 3.5.2)               
 jsonlite      1.6        2018-12-07 [1] CRAN (R 3.5.3)               
 knitr         1.22       2019-03-08 [1] CRAN (R 3.5.3)               
 magrittr      1.5        2014-11-22 [1] CRAN (R 3.5.3)               
 memoise       1.1.0      2017-04-21 [1] CRAN (R 3.5.3)               
 pillar        1.3.1      2018-12-15 [1] CRAN (R 3.5.3)               
 pkgbuild      1.0.3      2019-03-20 [1] CRAN (R 3.5.3)               
 pkgconfig     2.0.2      2018-08-16 [1] CRAN (R 3.5.3)               
 pkgload       1.0.2      2018-10-29 [1] CRAN (R 3.5.3)               
 prettyunits   1.0.2      2015-07-13 [1] CRAN (R 3.5.3)               
 processx      3.3.0      2019-03-10 [1] CRAN (R 3.5.3)               
 ps            1.3.0      2018-12-21 [1] CRAN (R 3.5.3)               
 purrr         0.3.2      2019-03-15 [1] CRAN (R 3.5.3)               
 R6            2.4.0      2019-02-14 [1] CRAN (R 3.5.3)               
 Rcpp          1.0.1      2019-03-17 [1] CRAN (R 3.5.3)               
 remotes       2.0.2      2018-10-30 [1] CRAN (R 3.5.3)               
 reprex        0.2.1      2018-09-16 [1] CRAN (R 3.5.3)               
 rlang         0.3.1      2019-01-08 [1] CRAN (R 3.5.3)               
 rmarkdown     1.12       2019-03-14 [1] CRAN (R 3.5.3)               
 rprojroot     1.3-2      2018-01-03 [1] CRAN (R 3.5.3)               
 rstudioapi    0.10       2019-03-19 [1] CRAN (R 3.5.3)               
 sessioninfo   1.1.1      2018-11-05 [1] CRAN (R 3.5.3)               
 sf            0.7-3      2019-02-21 [1] CRAN (R 3.5.3)               
 testthat      2.0.1      2018-10-13 [1] CRAN (R 3.5.3)               
 tibble        2.1.1      2019-03-16 [1] CRAN (R 3.5.3)               
 tidyselect    0.2.5      2018-10-11 [1] CRAN (R 3.5.3)               
 triebeard     0.3.0      2016-08-04 [1] CRAN (R 3.5.3)               
 units         0.6-2      2018-12-05 [1] CRAN (R 3.5.3)               
 urltools      1.7.2      2019-02-04 [1] CRAN (R 3.5.3)               
 usethis       1.4.0      2018-08-14 [1] CRAN (R 3.5.3)               
 utf8          1.1.4      2018-05-24 [1] CRAN (R 3.5.3)               
 whisker       0.3-2      2013-04-28 [1] CRAN (R 3.5.3)               
 withr         2.1.2      2018-03-15 [1] CRAN (R 3.5.3)               
 xfun          0.5        2019-02-20 [1] CRAN (R 3.5.3) 

[sckott]

thanks for your question. the method is here https://github.com/ropensci/crul/blob/master/R/paginator.R#L258-L279 - Looks like we have a cat("\n") in there right now. I think we could do a progress bar in the for loop we use in that page method

doing a revdep check right now on reverse deps, will check after thats done

[sckott]

can you reinstall remotes::install_github("ropensci/crul"), and try again. removed the "OK" message, and newlines, and now you can use a progress bar

[boshek]

Wow - this works perfectly. Thank you @sckott!

[sckott]

glad it works. sending to cran soon, so if you run into any problems with it get back to me asap