[REVIEW]: PyGModels: A Python package for exploring Probabilistic Graphical Models with Graph Theoretical Structures

[whedon]

Submitting author: @D-K-E (D. Kaan Eraslan)
Repository: https://github.com/D-K-E/graphical-models/
Version: v0.1.0
Editor: @dfm
Reviewer: @eigenfoo, @ankurankan
Archive: 10.5281/zenodo.4751740

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/1f3fb76e625510f42fb42602d5679e15"><img src="https://joss.theoj.org/papers/1f3fb76e625510f42fb42602d5679e15/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/1f3fb76e625510f42fb42602d5679e15/status.svg)](https://joss.theoj.org/papers/1f3fb76e625510f42fb42602d5679e15)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@eigenfoo & @ankurankan, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @dfm know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @eigenfoo

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@D-K-E) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @ankurankan

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@D-K-E) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @eigenfoo, @ankurankan it looks like you're currently assigned to review this paper 🎉.

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.42 s (822.3 files/s, 106916.9 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
HTML                           223            839           1981          29112
Python                          38           1191           2122           5919
CSS                              3            328             47           1627
JavaScript                      79            105            144           1618
Markdown                         2             59              0            169
YAML                             3              4              3             56
TeX                              1             25              0             26
-------------------------------------------------------------------------------
SUM:                           349           2551           4297          38527
-------------------------------------------------------------------------------


Statistical information for the repository '431a776d0fc99019b0d91060' was
gathered on 2021/03/16.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Qm Auber                        75         16160           5061          100.00

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Qm Auber                  11099           68.7          1.7               10.54

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.3150/08-BEJ172 is OK
- 10.1007/978-3-319-73235-0 is OK
- 10.1201/9780429463976-2 is OK
- 10.1111/1467-9868.00340 is OK
- 10.1017/9781108277495 is OK
- 10.1145/378886.380416 is OK

MISSING DOIs

- 10.25080/majora-7b98e3ed-001 may be a valid DOI for title: pgmpy: Probabilistic graphical models using python
- 10.2200/s00529ed1v01y201308aim023 may be a valid DOI for title: Reasoning with probabilistic and deterministic graphical models: exact algorithms
- 10.1007/978-3-642-17517-6_36 may be a valid DOI for title: Listing All Maximal Cliques in Sparse Graphs in Near-optimal Time

INVALID DOIs

- None

[dfm]

@eigenfoo, @ankurankan — This is the review thread for the paper. All of our communications will happen here from now on. Thanks again for agreeing to participate!

Please read the "Reviewer instructions & questions" in the first comment above.

Both reviewers have checklists at the top of this thread (in that first comment) with the JOSS requirements. As you go over the submission, please check any items that you feel have been satisfied. There are also links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention openjournals/joss-reviews#3115 so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for the review process to be completed within about 4-6 weeks but please make a start well ahead of this as JOSS reviews are by their nature iterative and any early feedback you may be able to provide to the author will be very helpful in meeting this schedule.

[dfm]

@D-K-E: Can you add those missing DOIs are reported above to the paper?

[eigenfoo]

Hello! I'm still in the middle of my review, but have one question so far.

From the README:

The primary goal is to facilitate the understanding of models and basic inference strategies using well documented data structures based only on Python 3 standard library.

As the overall library is not built for efficiency, we recommend not to use it in production. It should not be to difficult to transfer the concepts introduced in the source code though.

It seems that this repository is mainly written for pedagogical purposes, and perhaps not for actual use in scientific applications - in other words, it sounds like the library is meant to be read, and not run. @D-K-E am I correct in this? If so, @dfm can you advise if such a library is in scope for JOSS publication? I'm unsure if this library satisfies the "research software" requirement of JOSS - some clarity would be helpful.

---

Also noting from this thread (#3015 (comment)) that @Viva-Lambda is another account for @D-K-E, so the target repo definitely passes the "contribution and authorship" check box.

[D-K-E]

@dfm I am adding them right away.

@eigenfoo It is a little complicated issue. I am using this library for demonstrating how some data ¹ can be analyzed using probabilistic graphical models in my thesis so it is used at least in the case of one scientific work. But my focus during the writing of the library was to be as close as possible to definitions provided by text books (mostly Koller, Friedman 2009) so that there would be less friction between the text book and the implementation. The pedagogical aspect is more of an intended side effect which comes from being close to the text book. However the goal was to reproduce the definitions as close as possible not to teach them.

---

¹ historical documents annotated with a certain flavor of RDFa format.

[D-K-E]

@dfm added the requested DOIs, plus some changes in the documentation. I'll continue to improve the docs.

[dfm]

@eigenfoo: Great question! I am satisfied by your response @D-K-E, but the paper and documentation should make the research applications very clear and demonstrate where this package fits into the ecosystem. So perhaps the discussion in the README should be clarified with this in mind.

[eigenfoo]

@whedon generate pdf

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[eigenfoo]

@D-K-E this is great work! Detailed comments below.

---

Summary and Statements of Need

A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?

Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?

A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?

In both the documentation and the paper, the summary and statement are quite technical, does not seem to be for a non-specialist audience, and does not make it clear who the target audience is.

• Summary is too technical: what distinguishes this library from prior art is quite technical, but nevertheless I think it could be introduced in a friendlier way: I hope you don't mind if I break down point-by-point suggestions:

  1. PGMs are widely used
  2. However, most existing PGM libraries implement PGMs in a way that ignores their graphical nature (e.g. as explained on lines 64-66, "a factor is specified through an array of values", instead of a set of random variables and a function).
  3. PyGModel's value proposition is that it faithfully implements the graphical nature of PGMs, thereby giving PyGModel's instantiated objects both graph-theoretical and statistical properties, which allows users to explore and test inference algorithms that are rooted in both graph theory or statistics (paraphrasing lines 19-21)
  • @D-K-E would you say that I've summarized the paper correctly?

• Statements of need are unclear: the statements may be more technical than the summary, but after reading them I am still unsure who the intended users/audience is. If I'm correct in my previous summary, then the intended users are researchers exploring inference algorithms that depend on graph-theoretical properties of PGMs. PyGModel's lightweight nature, easy extensibility and faithful implementations of textbook algorithms are all good points, but become a lot clearer once I know that this library is for researchers!

---

Community Guidelines

Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Right now the README outlines how users might contribute to the software, but doesn't explicitly ask users who have issues/need support to file an issue. It's probably a safe bet from a user's perspective, but best to make this explicit!

---

Citations

From the paper, on lines 44-46:

PGMs are known for their wide range of applications in computer vision, information retrieval, disease diagnosis and more recently, in the context of our phd thesis, annotations of ancient documents.

It would be great to cite any portion of your PhD (or related work) that may already be published, especially if it showcases "real-world usage" of PyGModels. I understand if this isn't possible or if you may be unwilling though, so I'll go ahead and cross off the "references" checkbox.

---

Well Written

Overall, the paper is well-written: there are some minor grammatical mistakes (e.g. phd/PhD on line 45, depend/depends on line 62, not/no on line 67), but these errors don't get in the way of comprehension. I'm willing to let these errors slide, so I'll cross off the "quality of writing" checkbox.

[D-K-E]

@eigenfoo Well, first of all thank you for the suggestions, they really helped me to better phrase the problem.

Here is a list of changes that I made in accordance with your comments.

---

Summary

I don't think I could do better than your suggestions for the summary. I tried to incorporate them directly to the paper. It significantly decreased the length of the summary paragraph.

Statements of need

I added the following remark to the beginning of the section as per the request for the clarification of the intended public of the package:

Though the students of computer science or statistics might find a pedagogical
value going through source code along with a textbook on probabilistic
graphical models (something like Sucar [see @Sucar_2015] or Cowell [see
@Cowell_2005] or Koller and Friedman [see @Koller_Friedman_2009]), we believe
that the value proposition of PyGModels speaks mostly to researchers.

@dfm I hope that this clarifies how the packaged fits into the ecosystem, and resolves the discussion we had about the pedagogical use of it. I can add more details, if you find necessary.

---

Community Guidelines

I provided two templates for filling the docstrings of functions and explicitly asked users to file an issue with their doubt or intent in the README file.

---

Citations

Unfortunately published parts of my PhD does not involve the direct use of this library. I am hoping to present a paper at conference with results produced by this library, once I discuss things more with my supervisor. The problem is that there is not enough data (historical documents in the right format) to produce something immediately useful. Hence I need to either wait for experts to produce data, or find someway to transform already existing data to right format. In any case, I need to discuss it with my supervisor.

---

Well Written

As you might have noticed by now, English is not my first language, so there might be grammatical errors, and some awkward phrases here and there. I corrected the ones you pointed out. Feel free to make other suggestions regarding the language as well.

[ankurankan]

@whedon generate pdf

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[whedon]

👋 @ankurankan, please update us on how your review is going (this is an automated reminder).

[whedon]

👋 @eigenfoo, please update us on how your review is going (this is an automated reminder).

[ankurankan]

@D-K-E Quick question: I see that @Viva-Lambda has also contributed significantly to the repository but is not an author on the paper. Is there a reason for it or am I missing something?

Edit: Sorry, just noticed that you have already mentioned that both are your accounts.

[D-K-E]

@dfm All right I think I did what you said:
The new version is v0.1.0.
The release is called JOSS version.
DOI: 10.5281/zenodo.4751740

The title and the author are the same. Affiliation is okay as well. The manuscript is all good as well.

[dfm]

@whedon check references

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.25080/Majora-7b98e3ed-001  is OK
- 10.2200/S00893ED2V01Y201901AIM041 is OK
- 10.3150/08-BEJ172 is OK
- 10.1007/978-3-642-17517-6_36 is OK
- 10.1007/978-3-319-73235-0 is OK
- 10.1201/9780429463976-2 is OK
- 10.1111/1467-9868.00340 is OK
- 10.1017/9781108277495 is OK
- 10.1145/378886.380416 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[dfm]

@whedon set 10.5281/zenodo.4751740 as archive

[whedon]

OK. 10.5281/zenodo.4751740 is the archive.

[dfm]

@whedon set v0.1.0 as version

[whedon]

OK. v0.1.0 is the version.

[dfm]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.25080/Majora-7b98e3ed-001  is OK
- 10.2200/S00893ED2V01Y201901AIM041 is OK
- 10.3150/08-BEJ172 is OK
- 10.1007/978-3-642-17517-6_36 is OK
- 10.1007/978-3-319-73235-0 is OK
- 10.1201/9780429463976-2 is OK
- 10.1111/1467-9868.00340 is OK
- 10.1017/9781108277495 is OK
- 10.1145/378886.380416 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[whedon]

👋 @openjournals/joss-eics, this paper is ready to be accepted and published.

Check final proof 👉 openjournals/joss-papers#2302

If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#2302, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true

[dfm]

@eigenfoo, @ankurankan: Thanks again for your excellent and constructive reviews! I (and all of us at JOSS) really appreciate your contributions here.

@D-K-E: This is looking good! Thanks for your submission and your work on it throughout this process. I've handed this submission off the the Editors-in-Chief and they might have some final comments/edits before the final publication. But, in the meantime, congrats on your paper!

[D-K-E]

Thank you @dfm , and thanks @eigenfoo and @ankurankan for the constructive reviews.
Thanks @whedon the bot, for all your hard work.
Should I call the deposit=true, for me it looks good ?

[dfm]

@D-K-E: no, the Editor-in-chief will do that after their final checks.

[D-K-E]

All right thanks for the clarification

[danielskatz]

@whedon accept deposit=true

[whedon]

Doing it live! Attempting automated processing of paper acceptance...

[whedon]

🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦

[whedon]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

0. Check final PDF and Crossref metadata that was deposited 👉 Creating pull request for 10.21105.joss.03115 joss-papers#2305
1. Wait a couple of minutes, then verify that the paper DOI resolves https://doi.org/10.21105/joss.03115
2. If everything looks good, then close this review issue.
3. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? Notify your editorial technical team...

[eigenfoo]

Congrats on the published paper @D-K-E!

[danielskatz]

Congratulations to @D-K-E (D. Kaan Eraslan)!!

And thanks to @eigenfoo and @ankurankan for reviewing, and @dfm for editing!

[whedon]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.03115/status.svg)](https://doi.org/10.21105/joss.03115)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.03115">
  <img src="https://joss.theoj.org/papers/10.21105/joss.03115/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.03115/status.svg
   :target: https://doi.org/10.21105/joss.03115

This is how it will look in your documentation:

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://numfocus.org/donate-to-joss

[ankurankan]

Congratulations @D-K-E !!