[REVIEW]: Multi-Attribute Decision task builder for online experiments

[whedon]

Submitting author: @Yury-Shevchenko (Yury Shevchenko)
Repository: https://github.com/Yury-Shevchenko/mad
Version: 1.0.0
Editor: @alexhanna
Reviewer: @u01ai11
Archive: 10.5281/zenodo.3247311

Status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/30958288f2384aab953294c6b9cc9b73"><img src="http://joss.theoj.org/papers/30958288f2384aab953294c6b9cc9b73/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/30958288f2384aab953294c6b9cc9b73/status.svg)](http://joss.theoj.org/papers/30958288f2384aab953294c6b9cc9b73)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@u01ai11, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @alexhanna know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @u01ai11

Conflict of interest

• As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Version: Does the release version given match the GitHub release (1.0.0)?
• Authorship: Has the submitting author (@Yury-Shevchenko) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Authors: Does the paper.md file include a list of authors with their affiliations?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @u01ai11 it looks like you're currently assigned as the reviewer for this paper 🎉.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

👉 Check article proof 📄 👈

[alexhanna]

Hi @u01ai11. Just checking in on this.

[u01ai11]

Hi @u01ai11. Just checking in on this.

Will have this done soon. Just checking through the examples in the repo and trying to get them working.

[u01ai11]

This issue -- problems with opening the examples in Lab.js was resolved: Yury-Shevchenko/mad#3

[u01ai11]

In going through the examples, I can't find a way to edit the parameters in Open Lab, just Lab.js. The documentation should also mention that you need an account to view the example templates. Yury-Shevchenko/mad#4

[u01ai11]

The author has not made a release for the github repo

Yury-Shevchenko/mad#5

[u01ai11]

Cannot check the community guidelines box

Yury-Shevchenko/mad#6

[u01ai11]

Missing a doi for one paper in the paper.md file

Yury-Shevchenko/mad#7

[u01ai11]

I can't tick the checkbox, as there are no automated tests or steps outlined to verify the functioning of the software is as expected.
Given that this is a template for working on Lab.js and Open Lab, I am not sure what this would look like, or if this is necessary?

[u01ai11]

OK - I have completed the review for this.

I think this is a great tool for creating 'Mouselab' tasks quickly using an existing web-based builder (Lab.js) and a SaaS platform (Open Lab).

The documentation (in the README file) is clear, and the examples are helpful for understanding the functionality of the templates. One suggestion (rather than acceptance blocker) is to create a broken down by heading documentation using the GitHub's repo's wiki -- or some internal links on the README so you can skip to the relevant part of the doc easily.

Generally speaking, probably beyond the scope of acceptance criteria of JOSS, I am not sure where the line between complete 'Software' and a template experiment built in other software lies -- and where this particular submission falls?

But I do think this tool would benefit the research community who work using this particular type of task -- and as such, it should be published.

In summary, the things that need to be resolved for the checklist to be complete are:

• Reference DOI for one citation needs to be added.
• A release on Github needs to be made.
• Community guidelines for contributing etc need to be made.
• Some steps for verifying correct functioning need to be detailed, or @alexhanna agree that testing is not applicable in this instance.

[Yury-Shevchenko]

Dear Alex,
Thank you for your review and recognition of the merits of the multi-attribute decision builder.
To address your comments, I've added more information on how to open the task in Open Lab and lab.js in the documentation. Community guidelines have also been added.
As for your recommendation on wiki pages, this is definitely a good idea that I will follow in the future.
Concerning your comment on the boundary between software and a template, I understand your point of view, but I would also like to stress that the multi-attribute decision builder has its own independent functionality. It is about 700 lines of JavaScript code written to support the creation of a task with parameter values. So my point is that the builder is not just a template that was created in another program. Maybe the choice of the word "template" was a bit misleading, but the motivation behind it was to show a user that he/she can easily edit and customize the task.
I have updated the version on GitHub as well, and look forward to your feedback.
Best,
Yury

[u01ai11]

Dear Yury,

Thanks for making the changes I suggested. I have now been able to tick off all the boxes (other than the testing box) and as far as I am concerned, I would be pleased for it to be accepted in the current form.

Also, thanks for addressing my comment, it was more a point of thought rather than a criticism of the submission -- as I said before, I think this software will be very helpful for those building MAD tasks (especially those who are new to online research, or the type of task itself).

Best wishes,
Alex

[u01ai11]

@whedon set v1.0 as version

[whedon]

I'm sorry @u01ai11, I'm afraid I can't do that. That's something only editors are allowed to do.

[u01ai11]

whedon set v1.0 as version

I think this needs to be done, as the repo release was created!

[labarba]

👋 @alexhanna — Looks like this submission is ready for your final actions before publication.

[alexhanna]

Great. @Yury-Shevchenko, the last thing we need for this submission to make an archive in Zenodo/figshare/other service and update this thread with the DOI of the archive. For the Zenodo/figshare archive, please make sure that:

• The title of the archive is the same as the JOSS paper title
• That the authors of the archive are the same as the JOSS paper authors

[Yury-Shevchenko]

Thanks a lot @alexhanna! I have made an archive in Zenodo (https://zenodo.org/record/3247311#.XQdAPW8zaL9)- the DOI is 10.5281/zenodo.3247311

[alexhanna]

@whedon set 10.5281/zenodo.3247311 as archive

[whedon]

OK. 10.5281/zenodo.3247311 is the archive.

[alexhanna]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

OK DOIs

- 10.3758/s13428-014-0482-y is OK
- 10.3758/BF03195464 is OK
- 10.1016/j.actpsy.2018.02.007 is OK
- 10.1057/jors.1994.133 is OK

MISSING DOIs

- https://doi.org/10.1037/e722292011-085 may be missing for title: Identifying decision strategies in a consumer choice situation

INVALID DOIs

- None

[whedon]

Check final proof 👉 openjournals/joss-papers#767

If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#767, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true

[alexhanna]

@Yury-Shevchenko looks like one of the references is missing a DOI. Can you check on this?

[Yury-Shevchenko]

Dear @alexhanna I have not found the doi for this article. There is no doi on CrossRef or in other publications that cite this article. The article has been published in Judgment and Decision Making Journal (http://journal.sjdm.org/). The URL of the article is http://journal.sjdm.org/8801/jdm8801.html. I could not open the reference (https://doi.org/10.1037/e722292011-085) suggested by the search above.

[alexhanna]

@openjournals/joss-eics - what should happen in this case?

[kyleniemeyer]

@alexhanna whedon's DOI checker isn't perfect, so if manual inspection finds that there is no issue, then we can ignore it.

[alexhanna]

@kyleniemeyer got it. In this case, I think this one is ready for acceptance if everything else looks good on your end.

[danielskatz]

Will process soon. Thanks!

[danielskatz]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

OK DOIs

- 10.3758/s13428-014-0482-y is OK
- 10.3758/BF03195464 is OK
- 10.1016/j.actpsy.2018.02.007 is OK
- 10.1057/jors.1994.133 is OK

MISSING DOIs

- https://doi.org/10.1037/e722292011-085 may be missing for title: Identifying decision strategies in a consumer choice situation

INVALID DOIs

- None

[whedon]

Check final proof 👉 openjournals/joss-papers#774

If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#774, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true

[danielskatz]

👋 @u01ai11 - I note that you did not check off one of the items in the review checklist. Can you please check that off?

[danielskatz]

👋 @Yury-Shevchenko - I see a few small wording issues in the paper, which I have addressed in Yury-Shevchenko/mad#8

[Yury-Shevchenko]

Dear @danielskatz Thanks a lot for that! I have merged the changes.

[danielskatz]

👋 @u01ai11 - now I see your comments on the missing item - we'll go ahead with this given the particular type of software that this is

[danielskatz]

@whedon accept deposit=true

[whedon]

Doing it live! Attempting automated processing of paper acceptance...

[whedon]

🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦

[whedon]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

0. Check final PDF and Crossref metadata that was deposited 👉 Creating pull request for 10.21105.joss.01409 joss-papers#780
1. Wait a couple of minutes to verify that the paper DOI resolves https://doi.org/10.21105/joss.01409
2. If everything looks good, then close this review issue.
3. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? notify your editorial technical team...

[whedon]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](http://joss.theoj.org/papers/10.21105/joss.01409/status.svg)](https://doi.org/10.21105/joss.01409)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.01409">
  <img src="http://joss.theoj.org/papers/10.21105/joss.01409/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: http://joss.theoj.org/papers/10.21105/joss.01409/status.svg
   :target: https://doi.org/10.21105/joss.01409

This is how it will look in your documentation:

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: http://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://numfocus.salsalabs.org/donate-to-joss

[danielskatz]

Thanks @alexhanna for editing and @u01ai11 for reviewing

[Yury-Shevchenko]

Thanks a lot everyone!:)

[alexhanna]

Thanks everyone!