Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

download_zenodo(): use curl::multi_download() for parallel = TRUE; add unit tests #124

Merged
merged 9 commits into from
Nov 21, 2023

Conversation

florisvdh
Copy link
Member

This approach is faster than forking the R process, by using the libcurl facilities instead to do concurrent downloading of multiple files.

This PR also flips the default value for the parallel argument from FALSE to TRUE.

Also copying the unit tests from {n2khab} package in order to capture problems early-on.

@florisvdh
Copy link
Member Author

florisvdh commented Nov 20, 2023

@hansvancalster Two errors from {checklist} should now be fixed (a linter and the package version). However the other two errors seem false positives to me.

  • The update in man/convertdf_enc.Rd is just automated and is due to an upstream change in iconv() documentation.
  • Spell checks should not run in {testthat} snapshot files IMO, since these are generated automatically and contain code output and messages.

Copy link
Contributor

@hansvancalster hansvancalster left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I tested the new functionality and it works as expected!
I would however prefer a less stringent unit test that does not rely on a snapshot, but rather just check with expect_no_error() and expect_no_warning(). My comment below shows that snapshots may be a bit brittle.

tests/testthat/_snaps/zenodo.md Outdated Show resolved Hide resolved
@hansvancalster
Copy link
Contributor

hansvancalster commented Nov 21, 2023

  • The update in man/convertdf_enc.Rd is just automated and is due to an upstream change in iconv() documentation.

I think you should run checklist::check_documentation() locally and commit and push the change to this branch (make sure you have the correct roxygen2 installed 7.2.3).

  • Spell checks should not run in {testthat} snapshot files IMO, since these are generated automatically and contain code output and messages.

If you remove the snapshot (as I suggest above) this will no longer be an issue.

@florisvdh
Copy link
Member Author

@hansvancalster I relaxed the unit tests as requested.

  • The update in man/convertdf_enc.Rd is just automated and is due to an upstream change in iconv() documentation.

I think you should run checklist::check_documentation() locally and commit and push the change to this branch (make sure you have the correct roxygen2 installed 7.2.3).

I'm using latest versions. I cannot reproduce the GHA checklist error locally.

Output from checklist::check_documentation() is below (click to open):
> checklist::check_documentation()
ℹ Updating inborutils documentation
ℹ Loading inborutils
Spell checking settings

Root: /media/floris/DATA/git_repositories/inborutils 

Default language: en-GB 

NEWS.md                                        R/convertdf_enc.R                              
R/csv_to_sqlite.R                              R/data.R                                       
R/df_factors_to_char.R                         R/download_knmi_data_hour.R                    
R/download_zenodo.R                            R/extract_soil_map_data.R                      
R/gbif_species_name_match.R                    R/guess_crs.R                                  
R/inborutils.R                                 R/inborutils_defunct.R                         
R/inborutils_deprecated.R                      R/plot_coordinates_on_map.R                    
R/plot_label_splitter.R                        R/read_kmi_data.R                              
R/read_kml_file.R                              R/read_knmi_data.R                             
R/read_mow_data.R                              R/setup_codingclub_session.R                   
R/transform_coordinates.R                      R/variance_inflation.R                         
README.md                                      man/append_to_sqlite.Rd                        
man/convertdf_enc.Rd                           man/coordinate_example.Rd                      
man/csv_to_sqlite.Rd                           man/df_factors_to_char.Rd                      
man/download_knmi_data_hour.Rd                 man/download_zenodo.Rd                         
man/extract_soil_map_data.Rd                   man/gbif_species_name_match.Rd                 
man/get_name_gbif.Rd                           man/guess_crs.Rd                               
man/human_filesize.Rd                          man/inborutils-defunct.Rd                      
man/inborutils-deprecated.Rd                   man/inborutils-package.Rd                      
man/mow_header_split.Rd                        man/plot_coordinates_on_map.Rd                 
man/plot_label_splitter.Rd                     man/rain_knmi_2012.Rd                          
man/read_kmi_data.Rd                           man/read_kml_file.Rd                           
man/read_knmi_data.Rd                          man/read_mow_data.Rd                           
man/setup_codingclub_session.Rd                man/species_example.Rd                         
man/transform_coordinates.Rd                   man/vif.Rd                                     
tests/testthat.R                               tests/testthat/test-zenodo.R                   
tests/testthat/test_csv_to_sqlite.R            tests/testthat/test_gbif_species_name_match.R  
tests/testthat/test_setup_codingclub_session.R tests/testthat/test_transform_coordinates.R    
vignettes/data_loaders_examples.Rmd            vignettes/gbif_name_match.Rmd                  
vignettes/guess_crs.Rmd                        

Ignore patterns:

.github LICENSE.md

─ Session info ──────────────────────────────────────────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.3.2 (2023-10-31)
 os       Linux Mint 21.2
 system   x86_64, linux-gnu
 ui       RStudio
 language nl_BE:nl
 collate  nl_BE.UTF-8
 ctype    nl_BE.UTF-8
 tz       Europe/Brussels
 date     2023-11-21
 rstudio  2023.09.1+494 Desert Sunflower (desktop)
 pandoc   3.1.1 @ /usr/lib/rstudio/resources/app/bin/quarto/bin/tools/ (via rmarkdown)

─ Packages ──────────────────────────────────────────────────────────────────────────────────────────────────────
 ! package     * version date (UTC) lib source
   askpass       1.2.0   2023-09-03 [3] RSPM (R 4.3.0)
   assertable    0.2.8   2021-01-27 [3] RSPM (R 4.2.0)
   assertthat    0.2.1   2019-03-21 [3] CRAN (R 4.0.1)
   backports     1.4.1   2021-12-13 [3] RSPM (R 4.2.0)
   bit           4.0.5   2022-11-15 [3] RSPM (R 4.2.0)
   bit64         4.0.5   2020-08-30 [3] RSPM (R 4.2.0)
   blob          1.2.4   2023-03-17 [3] RSPM (R 4.2.0)
   cachem        1.0.8   2023-05-01 [3] RSPM (R 4.2.0)
   callr         3.7.3   2022-11-02 [3] RSPM (R 4.2.0)
   checklist     0.3.5   2023-11-21 [1] https://inbo.r-universe.dev (R 4.3.2)
   class         7.3-22  2023-05-03 [3] RSPM (R 4.2.0)
   classInt      0.4-10  2023-09-05 [3] RSPM (R 4.3.0)
   cli           3.6.1   2023-03-23 [3] RSPM (R 4.2.0)
   codemetar     0.3.5   2022-09-02 [3] RSPM (R 4.2.0)
   colorspace    2.1-0   2023-01-23 [3] RSPM (R 4.2.0)
   commonmark    1.9.0   2023-03-17 [3] RSPM (R 4.2.0)
   crayon        1.5.2   2022-09-29 [3] RSPM (R 4.2.0)
   credentials   2.0.1   2023-09-06 [3] RSPM (R 4.3.0)
   crosstalk     1.2.0   2021-11-04 [3] CRAN (R 4.1.2)
   curl          5.1.0   2023-10-02 [3] RSPM (R 4.3.0)
   cyclocomp     1.1.1   2023-08-30 [3] RSPM (R 4.3.0)
   data.table    1.14.8  2023-02-17 [3] RSPM (R 4.2.0)
   DBI           1.1.3   2022-06-18 [3] RSPM (R 4.2.0)
   desc          1.4.2   2022-09-08 [3] RSPM (R 4.2.0)
   devtools      2.4.5   2022-10-11 [3] RSPM (R 4.2.0)
   digest        0.6.33  2023-07-07 [3] RSPM (R 4.2.0)
   dplyr         1.1.3   2023-09-03 [3] RSPM (R 4.3.0)
   e1071         1.7-13  2023-02-01 [3] RSPM (R 4.2.0)
   ellipsis      0.3.2   2021-04-29 [3] CRAN (R 4.1.1)
   evaluate      0.23    2023-11-01 [3] RSPM (R 4.3.0)
   fansi         1.0.5   2023-10-08 [3] RSPM (R 4.3.0)
   fastmap       1.1.1   2023-02-24 [3] RSPM (R 4.2.0)
   fs            1.6.3   2023-07-20 [3] RSPM (R 4.2.0)
   generics      0.1.3   2022-07-05 [3] RSPM (R 4.2.0)
   gert          2.0.0   2023-09-26 [3] RSPM (R 4.3.0)
   ggplot2       3.4.4   2023-10-12 [3] RSPM (R 4.3.0)
   glue          1.6.2   2022-02-24 [3] RSPM (R 4.2.0)
   gtable        0.3.4   2023-08-21 [3] RSPM (R 4.2.0)
   hms           1.1.3   2023-03-21 [3] RSPM (R 4.2.0)
   htmltools     0.5.7   2023-11-03 [3] RSPM (R 4.3.0)
   htmlwidgets   1.6.2   2023-03-17 [3] RSPM (R 4.2.0)
   httpuv        1.6.12  2023-10-23 [3] RSPM (R 4.3.0)
   httr          1.4.7   2023-08-15 [3] RSPM (R 4.2.0)
   hunspell      3.0.3   2023-10-06 [3] RSPM (R 4.3.0)
 R inborutils  * 0.4.0   <NA>       [?] <NA>
   iterators     1.0.14  2022-02-05 [3] RSPM (R 4.2.0)
   jsonlite      1.8.7   2023-06-29 [3] RSPM (R 4.2.0)
   KernSmooth    2.23-22 2023-07-10 [3] RSPM (R 4.2.0)
   knitr         1.45    2023-10-30 [3] RSPM (R 4.3.0)
   later         1.3.1   2023-05-02 [3] RSPM (R 4.2.0)
   lazyeval      0.2.2   2019-03-15 [3] RSPM (R 4.2.0)
   leaflet       2.2.1   2023-11-13 [3] RSPM (R 4.3.0)
   lifecycle     1.0.4   2023-11-07 [3] RSPM (R 4.3.0)
   lintr         3.1.1   2023-11-07 [3] RSPM (R 4.3.0)
   lubridate     1.9.3   2023-09-27 [3] RSPM (R 4.3.0)
   magrittr      2.0.3   2022-03-30 [3] RSPM (R 4.2.0)
   memoise       2.0.1   2021-11-26 [3] RSPM (R 4.2.0)
   mime          0.12    2021-09-28 [3] RSPM (R 4.2.0)
   miniUI        0.1.1.1 2018-05-18 [3] RSPM (R 4.2.0)
   munsell       0.5.0   2018-06-12 [3] CRAN (R 4.0.1)
   oai           0.4.0   2022-11-10 [3] RSPM (R 4.2.0)
   openssl       2.1.1   2023-09-25 [3] RSPM (R 4.3.0)
   pillar        1.9.0   2023-03-22 [3] RSPM (R 4.2.0)
   pingr         2.0.2   2022-10-26 [3] RSPM (R 4.2.0)
   pkgbuild      1.4.2   2023-06-26 [3] RSPM (R 4.2.0)
   pkgconfig     2.0.3   2019-09-22 [3] CRAN (R 4.0.1)
   pkgdown       2.0.7   2022-12-14 [3] RSPM (R 4.2.0)
   pkgload       1.3.3   2023-09-22 [3] RSPM (R 4.3.0)
   plyr          1.8.9   2023-10-02 [3] RSPM (R 4.3.0)
   prettyunits   1.2.0   2023-09-24 [3] RSPM (R 4.3.0)
   processx      3.8.2   2023-06-30 [3] RSPM (R 4.2.0)
   profvis       0.3.8   2023-05-02 [3] RSPM (R 4.2.0)
   promises      1.2.1   2023-08-10 [3] RSPM (R 4.2.0)
   proxy         0.4-27  2022-06-09 [3] RSPM (R 4.2.0)
   ps            1.7.5   2023-04-18 [3] RSPM (R 4.3.0)
   purrr         1.0.2   2023-08-10 [3] RSPM (R 4.2.0)
   R6            2.5.1   2021-08-19 [3] RSPM (R 4.2.0)
   rcmdcheck     1.4.0   2021-09-27 [3] CRAN (R 4.1.1)
   Rcpp          1.0.11  2023-07-06 [3] RSPM (R 4.2.0)
   readr         2.1.4   2023-02-10 [3] RSPM (R 4.2.0)
   remotes       2.4.2.1 2023-07-18 [3] RSPM (R 4.2.0)
   renv          1.0.3   2023-09-19 [3] RSPM (R 4.3.0)
   rex           1.2.1   2021-11-26 [3] RSPM (R 4.2.0)
   rgbif         3.7.8   2023-09-11 [3] RSPM (R 4.3.0)
   rlang         1.1.2   2023-11-04 [3] RSPM (R 4.3.0)
   rmarkdown     2.25    2023-09-18 [3] RSPM (R 4.3.0)
   roxygen2      7.2.3   2022-12-08 [3] RSPM (R 4.2.0)
   rprojroot     2.0.4   2023-11-05 [3] RSPM (R 4.3.0)
   RSQLite       2.3.3   2023-11-04 [3] RSPM (R 4.3.0)
   rstudioapi    0.15.0  2023-07-07 [3] RSPM (R 4.2.0)
   scales        1.2.1   2022-08-20 [3] RSPM (R 4.2.0)
   sessioninfo   1.2.2   2021-12-06 [3] RSPM (R 4.2.0)
   sf            1.0-14  2023-07-11 [1] RSPM (R 4.3.1)
   shiny         1.7.5.1 2023-10-14 [3] RSPM (R 4.3.0)
   stringi       1.8.1   2023-11-13 [3] RSPM (R 4.3.0)
   stringr       1.5.1   2023-11-14 [3] RSPM (R 4.3.0)
   sys           3.4.2   2023-05-23 [3] RSPM (R 4.2.0)
   tibble        3.2.1   2023-03-20 [3] RSPM (R 4.3.0)
   tidyr         1.3.0   2023-01-24 [3] RSPM (R 4.2.0)
   tidyselect    1.2.0   2022-10-10 [3] RSPM (R 4.2.0)
   timechange    0.2.0   2023-01-11 [3] RSPM (R 4.2.0)
   tzdb          0.4.0   2023-05-12 [3] RSPM (R 4.2.0)
   units         0.8-4   2023-09-13 [3] RSPM (R 4.3.0)
   urlchecker    1.0.1   2021-11-30 [3] RSPM (R 4.2.0)
   usethis       2.2.2   2023-07-06 [3] RSPM (R 4.2.0)
   utf8          1.2.4   2023-10-22 [3] RSPM (R 4.3.0)
   vctrs         0.6.4   2023-10-12 [3] RSPM (R 4.3.0)
   whisker       0.4.1   2022-12-05 [3] RSPM (R 4.2.0)
   withr         2.5.2   2023-10-30 [3] RSPM (R 4.3.0)
   xfun          0.41    2023-11-01 [3] RSPM (R 4.3.0)
   xml2          1.3.5   2023-07-06 [3] RSPM (R 4.2.0)
   xopen         1.0.0   2018-09-17 [3] CRAN (R 4.0.1)
   xtable        1.8-4   2019-04-21 [3] CRAN (R 4.0.1)
   yaml          2.3.7   2023-01-23 [3] RSPM (R 4.2.0)

 [1] /home/floris/lib/R/library
 [2] /usr/local/lib/R/site-library
 [3] /usr/lib/R/site-library
 [4] /usr/lib/R/library

 R ── Package was removed from disk.

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────

###################################################################################################################
Checklist summary for the package located at:
/media/floris/DATA/git_repositories/inborutils
###################################################################################################################
1 allowed warning

-------------------------------------------------------------------------------------------------------------------
 documented but unexported functions: append_to_sqlite, get_name_gbif, inborutils, inborutils-defunct, inborutils-deprecated, inborutils-package, mow_header_split 
motivation: this is intended
###################################################################################################################
1 missing note

-------------------------------------------------------------------------------------------------------------------
 checking CRAN incoming feasibility ... NOTE
Maintainer: 'Hans Van Calster <hans.vancalster@inbo.be>'

New submission 
motivation: package will not be submitted to cran

###################################################################################################################

Don't forget to store changes with `write_checklist()`

###################################################################################################################

Summary from checklist::check_package():
###################################################################################################################
Checklist summary for the package located at:
/media/floris/DATA/git_repositories/inborutils
###################################################################################################################
1 allowed warning

-------------------------------------------------------------------------------------------------------------------
 documented but unexported functions: append_to_sqlite, get_name_gbif, inborutils, inborutils-defunct, inborutils-deprecated, inborutils-package, mow_header_split 
motivation: this is intended
###################################################################################################################
1 allowed note

-------------------------------------------------------------------------------------------------------------------
 checking CRAN incoming feasibility ... NOTE
Maintainer: 'Hans Van Calster <hans.vancalster@inbo.be>'

New submission 
motivation: package will not be submitted to cran
###################################################################################################################
R CMD check: 1 error
-------------------------------------------------------------------------------------------------------------------
 checking package dependencies ... ERROR
Package suggested but not available: ‘drat’

The suggested packages are required for a complete check.
Checking can be attempted without them by setting the environment
variable _R_CHECK_FORCE_SUGGESTS_ to a false value.

See section ‘The DESCRIPTION file’ in the ‘Writing R Extensions’
manual.

###################################################################################################################

Don't forget to store changes with `write_checklist()`

###################################################################################################################

Checking the package revealed some problems.

Copy link
Contributor

@hansvancalster hansvancalster left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! I pushed an update to hopefully let checklist pass.

@hansvancalster hansvancalster merged commit 1bfb740 into main Nov 21, 2023
5 checks passed
@hansvancalster hansvancalster deleted the download_multi branch November 21, 2023 12:20
@florisvdh
Copy link
Member Author

Yes, but a next round will rewrite the Rd file to its checklist-incompatible state (although note that I don't get the error from checklist as in GHA):

> checklist::check_documentation()
ℹ Updating inborutils documentationLoading inborutils
Writing convertdf_enc.Rd

@hansvancalster
Copy link
Contributor

Yes, but a next round will rewrite the Rd file to its checklist-incompatible state (although note that I don't get the error from checklist as in GHA):

> checklist::check_documentation()
ℹ Updating inborutils documentationLoading inborutils
Writing convertdf_enc.Rd

Not on my machine... So something is different. I wonder if it has something to do with different way in capitalization on linux vs windows. The change in a8781bf is just changing 'a' to 'A'.

@florisvdh
Copy link
Member Author

The change in a8781bf is just changing 'a' to 'A'.

I know 🙂 . ?iconv documentation has become lowercase for the first letter (check it out), which is why this changed here (it's documentation linked from iconv()). And indeed I also suspect on Windows this isn't captured / handled differently. Why it's a problem in GHA is puzzling (it runs in Linux), but I'm not eager to find out.

@florisvdh
Copy link
Member Author

Documentation mystery - could be just this:

  • you run R 4.3.0
  • GHA checklist workflow runs R 4.3.1 (in docker)
  • I run R 4.3.2 (current release)

Supposing the change in ?iconv occurred in 4.3.2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants