Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JOSS Review #19

Closed
Het-Shah opened this issue Sep 27, 2021 · 3 comments
Closed

JOSS Review #19

Het-Shah opened this issue Sep 27, 2021 · 3 comments

Comments

@Het-Shah
Copy link

Hi @a-ws-m! This will be an issue for my review for your JOSS submission.

Pros -

  • The API is straightforward for the end-user to use.

Cons -

  • The library seems to have ~1000 lines of actual code (1165). This begs the question about the substantial scholarly effort.
  • The paper lacks details about the library itself. It is unclear as to how the package is structured and what are the key functionalities.
  • It would be advantageous if you could make some basic ablations and provide their results. I am unaware of the datasets, but you can try on the smallest dataset and provide results.

Feel free to comment on these. I would be more than happy to discuss the same.

@a-ws-m
Copy link
Owner

a-ws-m commented Oct 25, 2021

Thanks for the review 👍. I've commented on the first con in the JOSS review thread as it was brought up by multiple reviewers. Regarding the second point, I've added a sentence to clarify some of the packages' features; let me know if there's some particular detail you think should be added, I intended to describe the key functionality within that paragraph. I'll work on adding some benchmarks now; there's a small difficulty in reporting the metrics because the most interesting ones (NLL) have no other values to compare against (to my knowledge) because current benchmarks in materials science don't report uncertainty quantification metrics.

a-ws-m added a commit that referenced this issue Jun 30, 2022
@a-ws-m
Copy link
Owner

a-ws-m commented Jun 30, 2022

The benchmarks are complete and I've reported the results for a few models with different hyperparameters in the appropriate results directory.

@Het-Shah
Copy link
Author

Great! everything seems fine to me now. I have completed the checklist on the JOSS thread and will cross-post this comment there as well.

@a-ws-m a-ws-m closed this as completed Jul 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants