[REVIEW]: COHESIVM: Combinatorial h+/e- Sample Investigation using Voltaic Measurements

[editorialbot]

Submitting author: @mxwalbert (Maximilian Wolf)
Repository: https://github.com/mxwalbert/cohesivm
Branch with paper.md (empty if default branch): 1-joss-review
Version: v1.0.0
Editor: @RMeli
Reviewers: @ericfell, @enricgrau
Archive: Pending

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/42b8ef67cb511833d81289249d273bd2"><img src="https://joss.theoj.org/papers/42b8ef67cb511833d81289249d273bd2/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/42b8ef67cb511833d81289249d273bd2/status.svg)](https://joss.theoj.org/papers/42b8ef67cb511833d81289249d273bd2)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@ericfell & @enricgrau, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review.
First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @RMeli know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @enricgrau

📝 Checklist for @ericfell

[editorialbot]

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

✅ OK DOIs

- 10.1002/anie.200603675 is OK
- 10.1039/D3MA00136A is OK
- 10.6028/jres.117.010 is OK
- 10.1063/5.0032116 is OK
- 10.21105/joss.06371 is OK
- 10.1080/08940886.2019.1608121 is OK

🟡 SKIP DOIs

- No DOI given, and none found for title: Materials Acceleration Platform: Accelerating Adva...
- No DOI given, and none found for title: Bayesian Optimization of Spray Parameters for the ...

❌ MISSING DOIs

- None

❌ INVALID DOIs

- None

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.90  T=0.10 s (1005.5 files/s, 287728.1 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
XML                              3             14              0          11151
Scheme                           2             26              0           9545
Python                          40           1022           1170           3917
Markdown                        20            217              0            742
reStructuredText                26            313            480            372
Jupyter Notebook                 4              0            326            113
TeX                              1              8              0             78
TOML                             1              4              2             51
Arduino Sketch                   1              5              0             50
YAML                             3              9             11             49
CSS                              1              7              0             38
INI                              2             13              0             27
-------------------------------------------------------------------------------
SUM:                           104           1638           1989          26133
-------------------------------------------------------------------------------

Commit count by author:

    29	mxwalbert
     3	selinawillswissen

[editorialbot]

Paper file info:

📄 Wordcount for paper.md is 407

✅ The paper includes a Statement of need section

[editorialbot]

License info:

✅ License found: MIT License (Valid open source OSI approved license)

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[enricgrau]

Review checklist for @enricgrau

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://github.com/mxwalbert/cohesivm?
• License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@mxwalbert) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[ericfell]

Review checklist for @ericfell

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://github.com/mxwalbert/cohesivm?
• License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@mxwalbert) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[enricgrau]

@mxwalbert Thank you for your submission to JOSS. I went through the paper and repository and I have concerns about the scholarly effort and necessity of the library. Here I list one by one:

1. Need and Novelty: the manuscript fails to show the problems in the field what the library attempts to solve. At the same time, it is not clear at all what the library brings to the table. The manuscript should clearly respond to the questions: What gaps exist in the field that this library fills? What software currently exists, if any, and why can’t they tackle those issue? How does this library compare to and trumps over those?
2. Functionality not mentioned: you make a good attempt on briefly mentioning the importance of accelerating materials research and combinatorial analysis. However, there is no mention on exactly how or what the library helps you do. The manuscript includes vague and generic phrases such as “simplify the process of…” and “…provides researchers with a with flexible and modular framework…”. The authors should aim to explain more specifically what these advantages are so the reader can more effectively understand if it is useful to them.
3. Insufficient scholarly effort: It is not clear to me that the library gathers enough work to be published. Even though the repository has existed for over a year, it only shows a couple of periods with activity and just a handful of commits. The fact that is has been used in previous work does not mean that there was enough effort behind the library. In this same regard, the work is not published or available as pre-print, so there is no way to even verify this.
4. Lis of authors: the main author has been the main contributor without a doubt, with just a couple of contributed lines by who I believe to be the second author. The other three authors have not committed any code, so it is unclear to me why they are listed as authors. I recommend using CRediT to assign the corresponding roles to each author. You can do this in the manuscript as a separate section at the end.

Because of these concerns I have not checked the library or tried the code at all. I’ll proceed to do so after the author successfully addresses these points. Please note that these points are trying to be constructive, you may have a good piece of software here but it is just no reflecting that. I’m happy to clarify and discuss all the above and help to address some of the points mentioned.

[mxwalbert]

@enricgrau Thank you for pointing out your concerns about our submission. I want to address them one by one:

1. Need and Novelty: the manuscript fails to show the problems in the field what the library attempts to solve. At the same time, it is not clear at all what the library brings to the table. The manuscript should clearly respond to the questions: What gaps exist in the field that this library fills? What software currently exists, if any, and why can’t they tackle those issue? How does this library compare to and trumps over those?

In the current manuscript, we attempt to answer these questions in the Statement of need section. We mention the purpose "(...)simplify the process of setting up and executing combinatorial voltaic measurements." and briefly compare the software against other existing tools in the second paragraph. Could you please concretize what exactly raised your concern?

2. Functionality not mentioned: you make a good attempt on briefly mentioning the importance of accelerating materials research and combinatorial analysis. However, there is no mention on exactly how or what the library helps you do. The manuscript includes vague and generic phrases such as “simplify the process of…” and “…provides researchers with a with flexible and modular framework…”. The authors should aim to explain more specifically what these advantages are so the reader can more effectively understand if it is useful to them.

We try to summarize the high-level functionality and advantages of the software in the first paragraph of the Statement of need section. However, we apparently fail to do so. The "vague and generic phrases" you mention are intentional because they specifically point out the advantages. We designed the software to be as abstract/generic as possible in order to make it usable for a large audience. Researchers should be able to use their measurement equipment with our software to run combinatorial measurements. Certainly, they still have to implement an API for their specific hardware to interface it with our software, but we provide an extensive guide in the documentation to do so. Then, the advantages will come into play because it's easy to setup and run experiments, store the data, collect relevant metadata and analyse the results. Furthermore, the implemented components can be reused which enables you to run a Measurement using different Devices. We intentionally left this level of detail out of the manuscript because it is written in the repository's readme. Do you think describing this briefly would resolve your concern?

3. Insufficient scholarly effort: It is not clear to me that the library gathers enough work to be published. Even though the repository has existed for over a year, it only shows a couple of periods with activity and just a handful of commits. The fact that is has been used in previous work does not mean that there was enough effort behind the library. In this same regard, the work is not published or available as pre-print, so there is no way to even verify this.

For this submission, the scholary effort should not be measured based on the number of commits. We do not have a background in software development and most of the work was done locally before even considering a publication. I think the effort behind this project becomes evident easily by looking at the source code and documentation. As for the cited previous work, we are currently drafting the paper but some effort can be verified by the checking the hardware documentation in the repository: MA8X8.

4. Lis of authors: the main author has been the main contributor without a doubt, with just a couple of contributed lines by who I believe to be the second author. The other three authors have not committed any code, so it is unclear to me why they are listed as authors. I recommend using CRediT to assign the corresponding roles to each author. You can do this in the manuscript as a separate section at the end.

The same reason from above applies here. But certainly, we will include a CRediT section in the revision to clarify the contributions.

Please let me know, if possible, how we should address concern 1 and if concerns 2-4 can be resolved as suggested in my comments.

[enricgrau]

@mxwalbert
For 1) and 2): I went over the manuscript again and I'm still not sure what the library does and why it exists. For instance, the first phrase in the Statement of need declares that it "... aims to simplify the process of setting up and executing combinatorial voltaic measurements." Why does this process need to be simplified? How does it make it more simple? Try to follow that sentence with "... by doing XYZ, which makes it simpler." or something along those ways. Another example is when you claim that it is simpler to use. How? Why? What makes other options complicated to implement? I'd recommend focusing on its specific functionality and being more specific overall. By doing that it will reflect its simplicity and other claims wihtout the need on emphazising so much on buzzwords. Maybe including something like Key features in the manuscript could help. Is COHESIVM for data colection? Data processing? Visalization? System control? All of the above? I'm genuinely not sure after reading the manuscript. In a few words, my issue is that I don't think the manuscript projects to be sufficiently useful or likely to be cited or used.

For 3): As per the JOSS guidelines, the number of commits is a factor we can use to judge scholarly effort. Any chance you can upload a draft to arXiv? That'd be enough to support the claim of the library being used before. Using the docs to support that is too much of self-sustained argument in my opinion (ie citing a paper in the same paper.)

[mxwalbert]

@enricgrau
Thank you for clarification. I will follow your suggestions to phrase a more specific description.

I agree that the number of commits can be used but it's not a requirement in my understanding. This is why I referred to the repository and documentation to base the judgement on them. Unfortunately, the draft is not ready yet, as we plan to finish it until end of November. However, the paper will be a completely separate contribution and only briefly mention the use of this software. So everything within the repository is open for judging the effort of this submission since it is not and will not be published elsewhere.

[mxwalbert]

@editorialbot set 1-joss-review as branch

[editorialbot]

Done! branch is now 1-joss-review

[mxwalbert]

@editorialbot generate pdf

@enricgrau

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[RMeli]

Hi everyone, how are things progressing? Could you please give me a quick update (just a few lines)? Thank you.

@enricgrau are you concerns being addressed? The number of commits should be taken with a pinch of salt, since it is somewhat arbitrary. The scholarly effort can be judged on the code itself, which in this case is ~4000 LOC (Python).

[mxwalbert]

Hi @RMeli,
the proposed changes were just recently implemented by us. We tried to address all raised issues and commited the changes in the 1-joss-review branch.

[ericfell]

Hi everyone, how are things progressing? Could you please give me a quick update (just a few lines)? Thank you.

@enricgrau are you concerns being addressed? The number of commits should be taken with a pinch of salt, since it is somewhat arbitrary. The scholarly effort can be judged on the code itself, which in this case is ~4000 LOC (Python).

Hi all, my comments have been addressed by the authors. Thanks to the authors for making their package open source.

[RMeli]

Thank you @ericfell for completing you review.

@enricgrau are your concerns being addressed? Could you please give a brief description of where we are at with your review? Many thanks.

[enricgrau]

The manuscript is somewhat changed and slightly improved, thus it is a bit more clear. I would insist, however, that the claim that it has been used before, citing an unpublished and unavailable manuscript, is no good. Unless @RMeli thinks this is fine, I cannot accept this. If @mxwalbert wants this claim, they need to upload a pre-print to arXiv, Research Gate, or similar.
The repo itself looks great. I only created a minute issue.

[mxwalbert]

Dear @enricgrau and @RMeli,
we can remove the unpublished work. It was only put there for future reference but since we now have the use-case description and linked tutorial, it's not needed anymore.

[mxwalbert]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[RMeli]

Thank you @mxwalbert for removing it and thank you @enricgrau for expressing your concerns and finalizing your review.

[RMeli]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[RMeli]

Hi @mxwalbert, sorry for the delay but this time of the year is super busy. I had a look at the paper and I think it's OK, but I feel you should remove the whole

So far, COHESIVM has been used for the investigation of oxide semiconductor heterojunctions where
it enables to quickly screen a matrix of 8 × 8 pixels on a single substrate
(25 mm × 25 mm)

sentence, or at least re-phrase it in a way that says how this application is described in the documentation. The link to the documentation is broken, and needs to be fixed.

(Ideally, you could put the previously referenced work on a pre-print server and cite it.)

[mxwalbert]

Dear @RMeli, no worries I was very busy as well! I re-phrased the mentioned sentence to make it less reference-dependent. To resolve the broken link, I merged the 1-joss-review to the main branch. (I had to clear my browser cache to be able to load the linked page.)

[RMeli]

Post-Review Checklist for Editor and Authors

Additional Author Tasks After Review is Complete

• Double check authors and affiliations (including ORCIDs)
• Make a release of the software with the latest changes from the review and post the version number here. This is the version that will be used in the JOSS paper.
• Archive the release on Zenodo/figshare/etc and post the DOI here.
• Make sure that the title and author list (including ORCIDs) in the archive match those in the JOSS paper.
• Make sure that the license listed for the archive is the same as the software license.

Editor Tasks Prior to Acceptance

• Read the text of the paper and offer comments/corrections (as either a list or a pull request)
• Check that the archive title, author list, version tag, and the license are correct
• Set archive DOI with @editorialbot set <DOI here> as archive
• Set version with @editorialbot set <version here> as version
• Double check rendering of paper with @editorialbot generate pdf
• Specifically check the references with @editorialbot check references and ask author(s) to update as needed
• Recommend acceptance with @editorialbot recommend-accept

[RMeli]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[RMeli]

@mxwalbert can you please proceed with the following?

• Make a release of the software with the latest changes from the review and post the version number here.
• Archive the release on Zenodo/figshare/etc and post the DOI here

Thank you.

[mxwalbert]

@RMeli

• v1.0.0-joss
• https://doi.org/10.5281/zenodo.14643235 (DOI maybe takes some time to be active: https://zenodo.org/records/14643235)

[RMeli]

Thanks @mxwalbert. However, you should do a proper release. v1.0.0-joss don't follow semantic versioning (and that's fine, but it is what you want to do?), but for example this new version is not reflected in pyproject.toml, which is stuck at 1.0.0.

[mxwalbert]

Thanks for the quick response @RMeli! Since there were no modifications in the code but only in the documentation, I kept the version semantically at 1.0.0. This should also indicate that it is the same as the version on PyPI. So for the installed package defined in pyproject.toml, there is practically no difference. Please let me know if this is fine for you, otherwise I can do a v1.0.1.

[RMeli]

Thanks for the additional context @mxwalbert, I missed that some of the changes were made before the release. I discussed this internally with other editors, and since the documentation is part of the codebase, we think it would be better to have a proper 1.0.1 release if it is not too much to ask. Would that be OK with you?

[RMeli]

Hi @mxwalbert, is there any progress on this? Thank you.

[mxwalbert]

Dear @RMeli, sorry for the delay. I will do the 1.0.1 release now and send you the updated Zenodo link.

[mxwalbert]

@RMeli

• v1.0.1
• https://doi.org/10.5281/zenodo.14747442