searchplos() returns different results when using internal pagination (limit>999)

[Bubblbu]

The same search returns different id values when limit > 999. Specifically, the returned DOIs are returned with appended references to sections in the paper. E.g. 10.1371/journal.pone.0030394/introduction, 10.1371/journal.pone.0030394/results_and_discussion, and the actual 10.1371/journal.pone.0030394.

Problem: The number of returned items is still the same as the found number of results without URL variations. ==> removing URL variations leads to missing articles in the final dataset

Examples:

No pagination:

  pub_dates = 'publication_date:[2016-01-01T00:00:00Z TO 2016-12-31T23:59:59Z]'
  journal = 'journal_key:PLoSONE'
  doc_type = 'doc_type:full'
  
  fl = 'id,publication_date,title,author'
  fq = list(journal, pub_dates, doc_type)

searchplos(q="*:*", fl=fl, fq=fq, limit=999)

# A tibble: 999 x 4
   id                           publication_date     author                                       title                                   
   <chr>                        <chr>                <chr>                                        <chr>                                   
 1 10.1371/journal.pone.0155491 2016-05-13T00:00:00Z Qian Yang,Yan Gu,Xuan Zhang,Jian-Mei Wang,Y… Uterine Expression of NDRG4 Is Induced …
 2 10.1371/journal.pone.0168631 2016-12-19T00:00:00Z Lei Deng,Wei Li,Xingming Yu,Chao Gong,Xueha… Correction: First Report of the Human-P…
 3 10.1371/journal.pone.0168627 2016-12-21T00:00:00Z Wen-Chan Huang,Hung-Lin Chen,Huan-Yuan Chen… Galectin-3 and Its Genetic Variation rs…
 4 10.1371/journal.pone.0155489 2016-05-12T00:00:00Z Yu-Mi Lee,Dong Eun Song,Tae Yong Kim,Tae-Yo… Risk Factors for Distant Metastasis in …
 5 10.1371/journal.pone.0168605 2016-12-21T00:00:00Z Zahirah Dhurmeea,Iker Zudaire,Emmanuel Chas… Reproductive Biology of Albacore Tuna (…
 6 10.1371/journal.pone.0168602 2016-12-16T00:00:00Z Elmarie Myburgh,Ryan Ritchie,Amy Goundry,Ke… Attempts to Image the Early Inflammator…
 7 10.1371/journal.pone.0168604 2016-12-15T00:00:00Z Ying Zhang,Tom Thomas,M J G Brussel,M F A M… Expanding Bicycle-Sharing Systems: Less…
 8 10.1371/journal.pone.0155469 2016-05-12T00:00:00Z Chia-Yi Tseng,Chin-Hung Lin,Lung-Yuan Wu,Jh… Potential Combinational Anti-Cancer The…
 9 10.1371/journal.pone.0168612 2016-12-16T00:00:00Z Igor Marchetti,Tom Loeys,Lauren B Alloy,Ern… Unveiling the Structure of Cognitive Vu…
10 10.1371/journal.pone.0155472 2016-05-12T00:00:00Z Chuanyu Yang,Charles A Powell,Yongping Duan… Deciphering the Bacterial Microbiome in…
# ... with 989 more rows

With pagination:

searchplos(q="*:*", fl=fl, fq=fq, limit=1000)

# A tibble: 1,000 x 4
   id                                                  publication_date     author                                                   title
   <chr>                                               <chr>                <chr>                                                    <chr>
 1 10.1371/journal.pone.0030394/introduction           2012-01-23T00:00:00Z Wei-Yao Wang,Tzong-Shi Chiueh,Jun-Ren Sun,Shin-Ming Tsa… NA   
 2 10.1371/journal.pone.0030394/results_and_discussion 2012-01-23T00:00:00Z Wei-Yao Wang,Tzong-Shi Chiueh,Jun-Ren Sun,Shin-Ming Tsa… NA   
 3 10.1371/journal.pone.0002157/materials_and_methods  2008-05-14T00:00:00Z Markus Pfenninger,Carsten Nowak                          NA   
 4 10.1371/journal.pone.0030394/supporting_information 2012-01-23T00:00:00Z Wei-Yao Wang,Tzong-Shi Chiueh,Jun-Ren Sun,Shin-Ming Tsa… NA   
 5 10.1371/journal.pone.0044137/materials_and_methods  2012-09-19T00:00:00Z Esmeralda Morillo,María Antonia Sánchez-Trujillo,José R… NA   
 6 10.1371/journal.pone.0113465/materials_and_methods  2014-12-17T00:00:00Z Patrick Durez,Pierre Vandepapeliere,Pedro Miranda,Antoa… NA   
 7 10.1371/journal.pone.0099112/introduction           2014-06-10T00:00:00Z Li Qi,Felix G Meinel,Chang Sheng Zhou,Yan E Zhao,U Jose… NA   
 8 10.1371/journal.pone.0099112/results_and_discussion 2014-06-10T00:00:00Z Li Qi,Felix G Meinel,Chang Sheng Zhou,Yan E Zhao,U Jose… NA   
 9 10.1371/journal.pone.0099112/materials_and_methods  2014-06-10T00:00:00Z Li Qi,Felix G Meinel,Chang Sheng Zhou,Yan E Zhao,U Jose… NA   
10 10.1371/journal.pone.0155488/title                  2016-05-20T00:00:00Z Jirayu Tanprasertsuk,Binxing Li,Paul S Bernstein,Rohini… NA   
# ... with 990 more rows

Session Info

Session info ------------------------------------------------------------------------------------------------------------------------------
 setting  value                       
 version  R version 3.4.4 (2018-03-15)
 system   x86_64, linux-gnu           
 ui       RStudio (1.1.453)           
 language en                          
 collate  en_US.UTF-8                 
 tz       America/Vancouver           
 date     2018-07-17                  

Packages ----------------------------------------------------------------------------------------------------------------------------------
 package    * version date       source        
 assertthat   0.1     2013-12-06 CRAN (R 3.3.1)
 cli          1.0.0   2017-11-05 CRAN (R 3.4.4)
 colorspace   1.3-2   2016-12-14 CRAN (R 3.3.2)
 crayon       1.3.4   2017-09-16 CRAN (R 3.4.2)
 crul         0.5.2   2018-02-24 CRAN (R 3.4.4)
 curl         3.2     2018-03-28 CRAN (R 3.4.4)
 DBI          0.5-1   2016-09-10 CRAN (R 3.3.1)
 devtools     1.12.0  2016-12-05 CRAN (R 3.3.2)
 digest       0.6.12  2017-01-27 CRAN (R 3.3.2)
 dplyr        0.5.0   2016-06-24 CRAN (R 3.3.1)
 ggplot2      2.2.1   2016-12-30 CRAN (R 3.3.2)
 gtable       0.2.0   2016-02-26 CRAN (R 3.3.1)
 jsonlite     1.5     2017-06-01 cran (@1.5)   
 lazyeval     0.2.0   2016-06-12 CRAN (R 3.3.1)
 lubridate    1.6.0   2016-09-13 CRAN (R 3.3.1)
 magrittr     1.5     2014-11-22 CRAN (R 3.3.1)
 memoise      1.0.0   2016-01-29 CRAN (R 3.3.2)
 munsell      0.4.3   2016-02-13 CRAN (R 3.3.1)
 pillar       1.2.2   2018-04-26 CRAN (R 3.4.4)
 plyr         1.8.4   2016-06-08 CRAN (R 3.3.1)
 R6           2.2.2   2017-06-17 cran (@2.2.2) 
 Rcpp         0.12.16 2018-03-13 CRAN (R 3.4.4)
 reshape2     1.4.2   2016-10-22 CRAN (R 3.3.2)
 rlang        0.2.0   2018-02-20 CRAN (R 3.4.4)
 rplos      * 0.8.0   2017-11-03 CRAN (R 3.4.4)
 rstudioapi   0.6     2016-06-27 CRAN (R 3.3.2)
 scales       0.4.1   2016-11-09 CRAN (R 3.3.2)
 solrium      1.0.0   2017-11-02 CRAN (R 3.4.4)
 stringi      1.1.2   2016-10-01 CRAN (R 3.3.2)
 stringr      1.2.0   2017-02-18 CRAN (R 3.3.2)
 tibble       1.4.2   2018-01-22 CRAN (R 3.4.4)
 triebeard    0.3.0   2016-08-04 cran (@0.3.0) 
 urltools     1.6.0   2016-10-17 cran (@1.6.0) 
 utf8         1.1.3   2018-01-03 CRAN (R 3.4.4)
 whisker      0.3-2   2013-04-28 CRAN (R 3.3.1)
 withr        2.1.2   2018-03-15 CRAN (R 3.4.4)
 xml2         1.1.1   2017-01-24 CRAN (R 3.3.2)
 yaml         2.1.19  2018-05-01 CRAN (R 3.4.4)

[sckott]

try after reinstalling. remotes::install_github("ropensci/rplos")

[Bubblbu]

Thanks! DOIs look fine now :)