[REVIEW]: An R reproducibility toolkit for the practical researcher

[editorialbot]

Submitting author: @paocorrales (Paola Corrales)
Repository: https://github.com/eliocamp/reproducibility-with-r/
Branch with paper.md (empty if default branch):
Version: v1.2.0
Editor: @yabellini
Reviewers: @Aariq, @luisDVA
Archive: Pending
Paper kind: learning module

Status

Status badge code:

HTML: <a href="https://jose.theoj.org/papers/a8d8bf5fd7a3b4d169121fedd9994f67"><img src="https://jose.theoj.org/papers/a8d8bf5fd7a3b4d169121fedd9994f67/status.svg"></a>
Markdown: [![status](https://jose.theoj.org/papers/a8d8bf5fd7a3b4d169121fedd9994f67/status.svg)](https://jose.theoj.org/papers/a8d8bf5fd7a3b4d169121fedd9994f67)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@Aariq, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review.
First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://openjournals.readthedocs.io/en/jose/reviewer_guidelines.html. Any questions/concerns please let @yabellini know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @Aariq

📝 Checklist for @luisDVA

[editorialbot]

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

✅ OK DOIs

- 10.1016/j.cogpsych.2010.05.004 is OK

🟡 SKIP DOIs

- No DOI given, and none found for title: The Turing Way: Sharing the Responsibility of Repr...
- No DOI given, and none found for title: Spaced Practice

❌ MISSING DOIs

- None

❌ INVALID DOIs

- None

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.90  T=0.84 s (557.7 files/s, 165788.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
CSV                              2              1              0          53770
HTML                           152          14984            166          22910
SVG                              7              1              3          19636
Sass                            98           1793            244          11009
Markdown                        78           1551              0           4131
JavaScript                      57            942            982           3426
XML                             26            156              0           1200
CSS                             21            139             76            789
TeX                              4             42             30            371
TOML                             4             33             64            305
R                                5             24            142             49
JSON                             5              0              0             45
YAML                             5              0              1             23
Rmd                              5            111            277              6
-------------------------------------------------------------------------------
SUM:                           469          19777           1985         117670
-------------------------------------------------------------------------------

Commit count by author:

   145	Elio Campitelli
    42	Pao
    28	Pao Corrales

[editorialbot]

Paper file info:

📄 Wordcount for paper.md is 1073

✅ The paper includes a Statement of need section

[editorialbot]

License info:

🔴 License found: Creative Commons Attribution 4.0 International (Not OSI approved)

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[Aariq]

Review checklist for @Aariq

Conflict of interest

• As the reviewer I confirm that I have read the JOSE conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• I confirm that I read and will adhere to the JOSE code of conduct.

General checks

• Repository: Is the source for this learning module available at the https://github.com/eliocamp/reproducibility-with-r/?
• License: Does the repository contain a plain-text LICENSE file with the contents of a standard license? (OSI-approved for code, Creative Commons for content)
• Version: Does the release version given match the repository release?
• Authorship: Has the submitting author (@paocorrales) made visible contributions to the module? Does the full list of authors seem appropriate and complete?

Documentation

• A statement of need: Do the authors clearly state the need for this module and who the target audience is?
• Installation instructions: Is there a clearly stated list of dependencies?
• Usage: Does the documentation explain how someone would adopt the module, and include examples of how to use it?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the module 2) Report issues or problems with the module 3) Seek support

Pedagogy / Instructional design (Work-in-progress: reviewers, please comment!)

• Learning objectives: Does the module make the learning objectives plainly clear? (We don't require explicitly written learning objectives; only that they be evident from content and design.)
• Content scope and length: Is the content substantial for learning a given topic? Is the length of the module appropriate?
• Pedagogy: Does the module seem easy to follow? Does it observe guidance on cognitive load? (working memory limits of 7 +/- 2 chunks of information)
• Content quality: Is the writing of good quality, concise, engaging? Are the code components well crafted? Does the module seem complete?
• Instructional design: Is the instructional design deliberate and apparent? For example, exploit worked-example effects; effective multi-media use; low extraneous cognitive load.

JOSE paper

• Authors: Does the paper.md file include a list of authors with their affiliations?
• A statement of need: Does the paper clearly state the need for this module and who the target audience is?
• Description: Does the paper describe the learning materials and sequence?
• Does it describe how it has been used in the classroom or other settings, and how someone might adopt it?
• Could someone else teach with this module, given the right expertise?
• Does the paper tell the "story" of how the authors came to develop it, or what their expertise is?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
  • The Turing Way is referenced with a link to a zenodo record rather than the DOI for that record. Might be better to use https://doi.org/10.5281/zenodo.2669548

These are great materials and the short paper introduces them well. I teach a workshop series that covers similar topics and I'm very excited to adopt some of the materials. I think the guide is written very thoughtfully with their target audience in mind. I especially appreciated the discussion of what reproducibility is and how it is not necessarily related to correctness and isn't a binary. As I was going through the materials on reproducibility.rocks, I opened a few minor issues:

• Update RStudio download link eliocamp/reproducibility-with-r#24
• next and previous page links don't go where I'd expect eliocamp/reproducibility-with-r#25
• link to Quarto journal article templates eliocamp/reproducibility-with-r#26
• RStudio & markdown bug resolved—update text? eliocamp/reproducibility-with-r#28

These issues mostly relate to content that is slightly out of date. I'll also note that while it is true that Quarto is still rapidly evolving, I think very soon (if not already) it will be worth it to teach Quarto instead of RMarkdown for reproducible scientific manuscripts and reports. RMarkdown and the rticles package certainly aren't going away anytime soon, but some of the frustrations I've encountered writing manuscripts in RMarkdown have been solved in Quarto—e.g. cross references to equations, images, code chunks, and other arbitrary sections—and I don't think Posit plans to actively develop RMarkdown going forwards. The authors may want to consider prioritizing an update to the reproducible reporting section in the next revision of these materials, although I do not think this update is necessary for acceptance of the manuscript.

[eliocamp]

Thanks for the review and the open issues. I've fixed 3/4. The missing one seems to be a limitation of the theme we're using for the website. I've open an issue in that repository to see what we can do.

And yes on quarto. We hadn't looked at the list of available journal templates for a while, so we didn't know it was so large (although not as large as rticles). We would need to look how to use a custom LaTeX template with quarto.

[yabellini]

@Aariq thank you so much for your review.

[yabellini]

@editorialbot add @luisDVA as reviewer

[editorialbot]

@luisDVA added to the reviewers list!

[yabellini]

Hello @luisDVA, thank you so much for agreeing to review this proposal 🙏 .

I added you as a reviewer. You can generate your checklist by calling our bot:

@ editorialbot generate my checklist

[paocorrales]

Hello! We'll be teaching the workshop this week and are making the usual checks and updates to the material. We haven't changed anything big, just broken links and small additions.
I wanted to mention it here in case anyone is currently visiting the web or the repo!

[luisDVA]

Review checklist for @luisDVA

Conflict of interest

• As the reviewer I confirm that I have read the JOSE conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• I confirm that I read and will adhere to the JOSE code of conduct.

General checks

• Repository: Is the source for this learning module available at the https://github.com/eliocamp/reproducibility-with-r/?
• License: Does the repository contain a plain-text LICENSE file with the contents of a standard license? (OSI-approved for code, Creative Commons for content)
• Version: Does the release version given match the repository release?
• Authorship: Has the submitting author (@paocorrales) made visible contributions to the module? Does the full list of authors seem appropriate and complete?

Documentation

• A statement of need: Do the authors clearly state the need for this module and who the target audience is?
• Installation instructions: Is there a clearly stated list of dependencies?
• Usage: Does the documentation explain how someone would adopt the module, and include examples of how to use it?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the module 2) Report issues or problems with the module 3) Seek support

Pedagogy / Instructional design (Work-in-progress: reviewers, please comment!)

• Learning objectives: Does the module make the learning objectives plainly clear? (We don't require explicitly written learning objectives; only that they be evident from content and design.)
• Content scope and length: Is the content substantial for learning a given topic? Is the length of the module appropriate?
• Pedagogy: Does the module seem easy to follow? Does it observe guidance on cognitive load? (working memory limits of 7 +/- 2 chunks of information)
• Content quality: Is the writing of good quality, concise, engaging? Are the code components well crafted? Does the module seem complete?
• Instructional design: Is the instructional design deliberate and apparent? For example, exploit worked-example effects; effective multi-media use; low extraneous cognitive load.

JOSE paper

• Authors: Does the paper.md file include a list of authors with their affiliations?
• A statement of need: Does the paper clearly state the need for this module and who the target audience is?
• Description: Does the paper describe the learning materials and sequence?
• Does it describe how it has been used in the classroom or other settings, and how someone might adopt it?
• Could someone else teach with this module, given the right expertise?
• Does the paper tell the "story" of how the authors came to develop it, or what their expertise is?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

Review

I enjoyed reviewing the contents and materials for this teaching module. The major topics are all highly relevant to reproducibility in scientific research, and the content for each day is laid out with a suitable pace and easy-to-follow exercises and examples. The course motivation and outline are also very well summarized in the corresponding short document.

I pointed out various minor fixes and made suggestions in the following issues:

• #31
• #32
• #33
• #34
• #35
• #36

Overall, the teaching module is cohesive, relevant, and with great potential for reuse. The other reviewer above already noted that some course elements that revolve around RMarkdown are at potential risk of becoming outdated or somewhat overlooked in favor of the Quarto publishing framework. This is already being addressed, but worth noting again.

The section on reporting with RMarkdown (day 1, part 3) is very well planned in terms of how additional topics and tools are introduced without overwhelming the learners with complexity. Since these materials have now been used to teach the course in repeated occasions, I wonder if there have been issues with LaTeX templates, and if so, how these were solved. If this is the case, perhaps this could be added to the instructor notes.

While I appreciate the power of using research compendia (day 2, part 3), this practice is not really widespread (at least within my field of biology/ecology/evolution). To get the most out of this tooling and approach, I suggest adding more explicit examples to the materials, and to dedicate some time to look at research compendia produced for published journal articles.

The materials for the fourth day are particularly valuable, given the clear and easy to follow instructions for setting up docker containers and even mapping folders from the virtual environments to the local machine. This content is rarely taught in research settings in biological science (the general target for the course), and the focus on R and running the IDE in the container is quite useful.

I appreciate the invitation to review and hope that the module authors find my comments useful.

[yabellini]

Thank you so much for you review @luisDVA !

Please @paocorrales and @eliocamp check Luis' comments and let's us know when you are ready.