[REVIEW]: ADaPT-ML: A Data Programming Template for Machine Learning

[whedon]

Submitting author: @nulberry (Andrea Whittaker)
Repository: https://github.com/U-Alberta/ADaPT-ML
Branch with paper.md (empty if default branch): main
Version: v1.0.0
Editor: @jmschrei
Reviewers: @aaronpeikert, @wincowgerDEV
Archive: 10.5281/zenodo.6402630

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/e846e08311cee3886d33101209166f4c"><img src="https://joss.theoj.org/papers/e846e08311cee3886d33101209166f4c/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/e846e08311cee3886d33101209166f4c/status.svg)](https://joss.theoj.org/papers/e846e08311cee3886d33101209166f4c)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@aaronpeikert & @wincowgerDEV, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @jmschrei know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @aaronpeikert

✨ Important: Please do not use the Convert to issue functionality when working through this checklist, instead, please open any new issues associated with your review in the software repository associated with the submission. ✨

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@nulberry) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @wincowgerDEV

✨ Important: Please do not use the Convert to issue functionality when working through this checklist, instead, please open any new issues associated with your review in the software repository associated with the submission. ✨

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@nulberry) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support #4076

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @aaronpeikert, @wincowgerDEV it looks like you're currently assigned to review this paper 🎉.

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Wordcount for paper.md is 612

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1007/s00778-019-00552-1 is OK
- 10.1145/3399579.3399867 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[whedon]

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.39 s (120.6 files/s, 9236.4 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          23            311            347           1627
Markdown                         4            137              0            476
YAML                             8             48             10            347
Bourne Shell                     5             21              1             94
TeX                              1              5              0             74
Dockerfile                       3             15              4             69
XML                              1              0              0             12
JSON                             2              0              0              2
-------------------------------------------------------------------------------
SUM:                            47            537            362           2701
-------------------------------------------------------------------------------


Statistical information for the repository '5561e150ca19ae9ceb66ea2c' was
gathered on 2022/01/07.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Andrea Whittaker               175          6151           3860          100.00

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Andrea Whittaker           2291           37.2          3.1                5.85

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[jmschrei]

Howdy @aaronpeikert and @wincowgerDEV

Thanks for agreeing to review this submission.

The process for conducting a review is outlined above in the checklist from @whedon. Please check the boxes during your review to keep track, as well as make comments in this thread or open issues in the repository itself to point out issues you encounter. Keep in mind that our aim is to improve the submission to the point where it is of high enough quality to be accepted, rather than to provide a yes/no decision, and so having a conversation with the authors is encouraged rather than providing a single review post at the end of the process.

Here are the review guidelines: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html
And here is a checklist, similar to above: https://joss.readthedocs.io/en/latest/review_checklist.html

Please let me know if you encounter any issues or need any help during the review process, and thanks for contributing your time to JOSS and the open source community!

[whedon]

👋 @aaronpeikert, please update us on how your review is going (this is an automated reminder).

[whedon]

👋 @wincowgerDEV, please update us on how your review is going (this is an automated reminder).

[wincowgerDEV]

👋 @wincowgerDEV, please update us on how your review is going (this is an automated reminder).

I have submitted several issues to the repo I am reviewing and currently cannot get the software running on my machine so that is preventing me from progressing on my review but hopefully that will be remedied soon.

[jmschrei]

Howdy @aaronpeikert and @wincowgerDEV, how is the review going?

[wincowgerDEV]

Thanks for the ping. I didn't realize that the software was only fully fleshed out for Mac and Linux devices when I started because it wasn't listed in the documentation materials. I don't have a Mac or Linux device so I'm not sure if I should build a virtual machine to do the task or I should request better documentation for Windows which the maintainer says should be possible, though it is not fully fleshed out. We could also try and add another reviewer who does have a Linux device and I can end my review at this time with all the components done besides the beta test. Warm Regards Win

…

On Sun, Feb 20, 2022, 10:31 PM Jacob Schreiber ***@***.***> wrote: Howdy @aaronpeikert <https://github.com/aaronpeikert> and @wincowgerDEV <https://github.com/wincowgerDEV>, how is the review going? — Reply to this email directly, view it on GitHub <#4038 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGMUJU74FB6322W5QM2HWFTU4HL5DANCNFSM5LNTYIUQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[aaronpeikert]

Hi, thanks for the reminder. I am a Linux user, but nevertheless had troubles with the setup that kinda dragged me down and made me forget that I still have to review. So I just needed a reminder to get back to it.

[wincowgerDEV]

Aaron, that's good to know. It sounds to me like the documentation isn't quite ready for prime time use yet and we should create an issue in the repo to outline some of the things that are still left undone from a user perspective because I don't think users would do the level of debugging that we are doing right now to get it to work. There needs to be a hold-your-hand level tutorial that gets everything up and running. I'll start the issue shortly. I'll stay focused on the windows aspects for now since there isn't a tutorial for using this with VMs. Though another thing the maintainer could provide is an AMI or something to get this spooled up immediately.

…

On Mon, Feb 21, 2022, 7:20 AM Aaron Peikert ***@***.***> wrote: Hi, thanks for the reminder. I am a Linux user, but nevertheless had troubles with the setup that kinda dragged me down and made me forget that I still have to review. So I just needed a reminder to get back to it. — Reply to this email directly, view it on GitHub <#4038 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGMUJU5JY262KJV4W2INNVLU4JJ2RANCNFSM5LNTYIUQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[aaronpeikert]

Hi Win,
it is surely a good idea to improve Windows support. My issues were related to some dependency and not the project itself. My Linux setup is also not too common (only a few ten thousand users I think). I will test the project under some normal Linux and possibly under Mac OS and report back. My fear is that docker-compose is not as portable as advertised (I had the weirdest issues getting a container up and running somewhere else). A tricky problem for @U-Alberta to address. Having a virtual machine solution available might work, but still can be hard for the target user group. Let's see how widespread the problem actually is, since documentation can only address common problems.

[jmschrei]

Knowing that it's difficult to use the software is important. I agree that they should improve the documentation to make it clear what platforms are fully supported. I don't think it's necessary that they make it work on all platforms if that's particularly difficult. If one of you does not have access to the supported platforms I think it's okay for your review to be higher level, e.g., at the documentation and organizational level and on the paper itself. But it would be great if at least one of you could get the code working on a standard platform, at least in a minimal format.

[wincowgerDEV]

Thanks for the input Jacob. I'll see if the author has an interest in broadening the support for Windows and if not will see if Aaron's able to get it working on the options they have.

…

On Mon, Feb 21, 2022, 10:59 AM Jacob Schreiber ***@***.***> wrote: Knowing that it's difficult to use the software is important. I agree that they should improve the documentation to make it clear what platforms are fully supported. I don't think it's necessary that they make it work on all platforms if that's particularly difficult. If one of you does not have access to the supported platforms I think it's okay for your review to be higher level, e.g., at the documentation and organizational level and on the paper itself. But it would be great if at least one of you could get the code working on a standard platform, at least in a minimal format. — Reply to this email directly, view it on GitHub <#4038 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGMUJUZPC36TFILBRVHB4MDU4KDS5ANCNFSM5LNTYIUQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[nulberry]

Hello everyone,
Thank you for pointing out the issues with the documentation for Windows. I apologize for not having this documentation clear and accurate at the beginning of this review, and for any delay or frustration this may have caused. The latest commit has the improved documentation.

Hi Win, it is surely a good idea to improve Windows support. My issues were related to some dependency and not the project itself. My Linux setup is also not too common (only a few ten thousand users I think). I will test the project under some normal Linux and possibly under Mac OS and report back. My fear is that docker-compose is not as portable as advertised (I had the weirdest issues getting a container up and running somewhere else). A tricky problem for @U-Alberta to address. Having a virtual machine solution available might work, but still can be hard for the target user group. Let's see how widespread the problem actually is, since documentation can only address common problems.

@aaronpeikert , I'm understanding from this comment that you are experiencing new issues that I am unaware of that are making it difficult for you to complete your review, is this correct? In U-Alberta/ADaPT-ML#9 we addressed the issue caused by CrateDB failing the bootstrap checks, and you commented that the tests ran successfully. Can you please open a new issue detailing the problems you are having with docker-compose, and if you've determined how widespread the problem is?

[aaronpeikert]

@aaronpeikert , I'm understanding from this comment that you are experiencing new issues that I am unaware of that are making it difficult for you to complete your review, is this correct?

No, on my personal machine, everything is fine. Nevertheless, I wanted to test your software on another Linux machine, on Mac, and on Windows (if you plan on supporting Windows). Just to verify that people can follow the documentation and do not run into any issues.

[wincowgerDEV]

@aaronpeikert I am running through the Windows set up so no need to do that on your part.

…

On Fri, Feb 25, 2022 at 2:52 AM Aaron Peikert ***@***.***> wrote: @aaronpeikert <https://github.com/aaronpeikert> , I'm understanding from this comment that you are experiencing new issues that I am unaware of that are making it difficult for you to complete your review, is this correct? No, on my personal machine, everything is fine. Nevertheless, I wanted to test your software on another Linux machine, on Mac, and on Windows (if you plan on supporting Windows). Just to verify that people can follow the documentation and do not run into any issues. — Reply to this email directly, view it on GitHub <#4038 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGMUJU227JAWEDC6CXYUC5LU45NONANCNFSM5LNTYIUQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

-- ´¯`·.¸¸.·´¯`·.´¯`·.¸¸.·´¯`ツ ------------------------------ --------------------- *Win Cowger, PhD * Pronouns: he/him Research Scientist Moore Institute for Plastic Pollution Research *Contact Info* 515-298-3869 | ***@***.*** | @Win_OpenData <https://twitter.com/Win_OpenData> *Websites* Personal Website: www.wincowger.com Currently Employed: https://mooreplasticresearch.org/ Alumni Of: https://www.thegraylab.org/ Project Websites: www.openspecy.org Research Gate: https://www.researchgate.net/profile/Win-Cowger Github: https://github.com/wincowgerDEV OSF: https://osf.io/kxeh5/ ------------------------------------------------------------------------------------

[nulberry]

To fully integrate Windows and MacOS into my CI, I am working on setting up windows-2022 and macos-latest runners to perform the same testing that was running on ubuntu-latest. I have only basic experience with GitHub Workflows so it might take another day or two.

[wincowgerDEV]

@aaronpeikert @nulberry @jmschrei, My review is complete, the software is working as expected and I have reviewed the rest of the tasks, looking forward to seeing this published.

[nulberry]

This is good to know, thanks. I've deleted and redone the release, and now Zenodo has created a DOI: https://zenodo.org/badge/latestdoi/380004451

[jmschrei]

@editorialbot set 10.5281/zenodo.6402630 as archive

[editorialbot]

Done! Archive is now 10.5281/zenodo.6402630

[jmschrei]

@editorialbot recommend-accept

[editorialbot]

Attempting dry run of processing paper acceptance...

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1007/s00778-019-00552-1 is OK
- 10.1145/3399579.3399867 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[editorialbot]

👋 @openjournals/joss-eics, this paper is ready to be accepted and published.

Check final proof 👉 openjournals/joss-papers#3109

If the paper PDF and the deposit XML files look good in openjournals/joss-papers#3109, then you can now move forward with accepting the submission by compiling again with the command @editorialbot accept

[kyleniemeyer]

Hi @nulberry, could you edit the metadata of the Zenodo archive so that your name is spelled out (rather than using your GitHub username)? After that we should be ready to publish.

[nulberry]

Hi! I created a .zenodo.json file in the repo to define the metadata and replaced nulberry with my full name. Is that all I have to do, or do I have to flip the switch on the Zenodo archive in order for it to apply my changes?

[kyleniemeyer]

@nulberry you can actually just log into Zenodo and edit the metadata for that record

[nulberry]

Done! :)

[kyleniemeyer]

@editorialbot accept

[editorialbot]

Doing it live! Attempting automated processing of paper acceptance...

[editorialbot]

🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦

[editorialbot]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

0. Check final PDF and Crossref metadata that was deposited 👉 Creating pull request for 10.21105.joss.04038 joss-papers#3110
1. Wait a couple of minutes, then verify that the paper DOI resolves https://doi.org/10.21105/joss.04038
2. If everything looks good, then close this review issue.
3. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? Notify your editorial technical team...

[kyleniemeyer]

Congratulations @nulberry on your article's publication in JOSS!

Many thanks to @aaronpeikert and @wincowgerDEV for reviewing this, and @jmschrei for editing.

[editorialbot]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.04038/status.svg)](https://doi.org/10.21105/joss.04038)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.04038">
  <img src="https://joss.theoj.org/papers/10.21105/joss.04038/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.04038/status.svg
   :target: https://doi.org/10.21105/joss.04038

This is how it will look in your documentation:

We need your help!

The Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://numfocus.org/donate-to-joss