[REVIEW]: PAM: Population Activity Modeller

[editorialbot]

Submitting author: @fredshone (Fred Shone)
Repository: https://github.com/arup-group/pam
Branch with paper.md (empty if default branch): joss
Version: v0.3.2
Editor: @martinfleis
Reviewers: @jamesdamillington, @martibosch
Archive: 10.5281/zenodo.10948231

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/c8737f0649d60fffcd3ba0aa25fa5f40"><img src="https://joss.theoj.org/papers/c8737f0649d60fffcd3ba0aa25fa5f40/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/c8737f0649d60fffcd3ba0aa25fa5f40/status.svg)](https://joss.theoj.org/papers/c8737f0649d60fffcd3ba0aa25fa5f40)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@jamesdamillington & @martibosch, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review.
First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @martinfleis know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @jamesdamillington

📝 Checklist for @martibosch

[editorialbot]

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.88  T=0.29 s (642.4 files/s, 198646.1 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                         103           4789           2487          18491
JSON                             5              0              0           4770
XML                             17            182             39           4143
Jupyter Notebook                18              0          17653           2387
Markdown                        16            313              0            757
YAML                            14             37              8            478
SQL                              3              0              0            164
XSD                              1              8              2            125
DTD                              4             89            254            102
TOML                             1             24             14             82
TeX                              1              7              0             77
HTML                             1              3              1              9
CSS                              1              1              2              7
Dockerfile                       1              2              0              7
-------------------------------------------------------------------------------
SUM:                           186           5455          20460          31599
-------------------------------------------------------------------------------


gitinspector failed to run statistical information for the repository

[editorialbot]

Wordcount for paper.md is 714

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.5334/baw is OK
- 10.17226/22357 is OK
- 10.1080/12265934.2013.835118 is OK
- 10.14279/depositonce-9835 is OK
- 10.1016/j.procs.2021.03.089 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[martinfleis]

👋🏼 @fredshone, @jamesdamillington, @martibosch this is the review thread for the paper. All of our communications will happen here from now on.

All reviewers should create checklists with the JOSS requirements using the command @editorialbot generate my checklist. As you go over the submission, please check any items that you feel have been satisfied. There are also links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues (and small pull requests if needed) on the software repository. When doing so, please mention https://github.com/openjournals/joss-reviews/issues/6097 so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

As agreed, I expected that the review will take a bit longer than usual, aiming at January. Please let me know if any of you require significantly more time at any point. We can also use editorialbot to set automatic reminders if you know you'll be away for a known period of time.

Please feel free to ping me (@martinfleis) if you have any questions/concerns.

Thanks!

[martinfleis]

Hi @jamesdamillington, @martibosch, just a minor reminder that this review is ongoing. We agreed to target end of January so there's still a bit of time but wanted to ensure it won't slip of your radar. Thanks!

[jamesdamillington]

Review checklist for @jamesdamillington

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://github.com/arup-group/pam?
• License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@fredshone) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[jamesdamillington]

I have now reviewed this. It looks very good and as shown by the checklist I am happy it meets all the criteria and could be accepted. The tool works as expected and from the examples I have explored I believe the functional claims are supported.

One minor point (regarding the paper, not the code or implementation, so I will make it here) is that Figure 1 is not referenced in the main text nor is it explained anywhere I can see (neither the paper nor in the documentation). The figure is an illustration to the text, to demonstrate some point made in the text etc., so should be referenced. Furthermore, while the figure may be self-explanatory to the authors but it may not be to the reader. For example, I am not clear what the diagonal shading between work and shop for Persons A and B means. I assume A and B work at the same location given the brown/pink shading but I don't understand the diagonal connections. I think some general description of Figure 1 plus maybe explanation of this point would be useful (in the paper at least, but then maybe also copy the text where the Figure appears in documentation).

[fredshone]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[jamesdamillington]

Thanks for editing as suggested @fredshone Looks good.

[fredshone]

@jamesdamillington many thanks 🙏🏻. I added a reference for the offending figure in the text and some explanation to the caption. It is a rather abstract representation of a complex data structure - please let me know if you'd like the figure reworked or removed.

[martinfleis]

Hey @martibosch, could you give us an estimation of when you'd be able to start? I know you mentioned you are a bit swamped but wanted to ensure we'll be able to do the review in a timely manner. Thanks!

@jamesdamillington Thanks a lot!

[martinfleis]

@editorialbot remind @martibosch in one week

[editorialbot]

Reminder set for @martibosch in one week

[editorialbot]

👋 @martibosch, please update us on how your review is going (this is an automated reminder).

[martinfleis]

Just FYI, I had a short email exchange with @martibosch who needs a bit more time for the review.

[martibosch]

Review checklist for @martibosch

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://github.com/arup-group/pam?
• License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@fredshone) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[martibosch]

Hello,

first of all sorry for the delay in my review. Here we go:

This is a super solid package which provides a super friendly/Pythonic interface for population activity modeling. A few years ago I was interested in activity-based population synthesis to model the impacts of the spatial organization of urban areas on travel behaviour, but I was overwhelmed due to the lack of existing Python tools. I think this library could have changed that.

I like that there are so many notebooks and that they provide a great overview of the functionalities - again, I find the library code very intuitive and the notebooks are very easy to follow. Regarding the library itself, I would only like to raise three points:

1. The fact that it is so Pythonic and object oriented makes me wonder about performance (which you point out in the manuscript). All the notebooks show small populations, which are great for pedagogical purposes but how long would it take to perform common tasks for a population with, e.g., 1M households? For instance, in 07_travel_survey_to_matsim.ipynb, what is the size of the data, e.g., number of households, people, trips...? how long does it take to run the notebook? Overall, it would be interesting to see more real-world examples.
2. The data structures of 06_travel_plots.ipynb remind me of scikit-mobility, I wonder if it would be interesting to mention the links or think of a more seamless integration for this part (please do not hesitate to point out if I am mistaken and there is no point in doing so).
3. (tiny detail, feel free to ignore) Cloning the repo took some time: it uses 25 MB (even though 217 MB were downloaded, I do not know why this happens but I suppose is more related to git/github and thus out of the scope of this review) - I wonder if it would be better to put the examples in a separate repo, but this is completely up to the maintainers.

Finally, I have some manuscript corrections:

1. The name for the last author is missing (after the "and").
2. I believe that "Python" should always be capitalized.
3. lines 23-25 and 28-29 are repeated. I believe a different (synonym) sentence should be used in the latter.
4. The biggest issue I see is that the summary does not read as such: a summary should provide an overview of the problem as well as a brief overview of how the proposed tool addresses it. The latter is clearly missing.

Martí

[fredshone]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[fredshone]

Thank you for the thorough review @martibosch.

The revisions in the above paper should address your manuscript corrections:

• fixed missing author
• capitalized Python
• removed repetition
• added a beginning and end paragraph to summary

Regarding the project:

1. Performance

• Certainly restricted by the OO design, but we trade for use-ability and broad applicability.
• A millions agents is about 10G. Read time from tabular or xml is about 5 minutes.
• There has been profiling and reworking of very commonly used features (such as io) to speed them up.
• Functionality in the CLI uses streaming to minimize memory requirements. It is possible to built some parallelization on this streaming approach for appropriate tasks.
• The examples are deliberately minimal as we include them (and their data) in our repo and CI pipeline. This ensures we find out when they break. I don't think it is feasible for small open source projects to long-term maintain examples that use external data. Happy to talk anyone's ear off about this 😃 .
• We do periodically muse on performance and our next option would be to be stricter about data types and write some of the core objects in C/Rust. But this is some long way off.

2. Scikit-mobility:

• Thanks for the link. I agree there is some duplicate functionality.
• I have raised this issue and will discuss with team next week.

3. Repo bloat:

• Thanks for bringing this up. In-fact the project is 250M which is unnecessary.
• Normally a download size (217M) would be compressed and significantly smaller than uncompressed rep size.
• In fact we don't get much compression because we have a bloated git history.
• I have detailed in this issue and will push for a clean up next week.
• Pretty sure the bulk of the bloat is some big data files lurking in our history.
• We actually have checks for this in our pipeline and pre-commit so don't know how these got through.

Many thanks!

[fredshone]

@editorialbot accept

[editorialbot]

I'm sorry @fredshone, I'm afraid I can't do that. That's something only eics are allowed to do.

[fredshone]

(pdf and xml look correct)

[oliviaguest]

@fredshone can you change the reference to be without the double brackets here please:

(e.g., (Castro et al., 2023)).

[oliviaguest]

see: arup-group/pam#273

[fredshone]

@oliviaguest merged (thanks for the PR!)

[fredshone]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[oliviaguest]

@fredshone sorry, I should have clarified! That was one, but you have more. Can you check and edit them, please?

[fredshone]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[fredshone]

@oliviaguest my bad - too excited to get a PR 🥇 - I found one other (Shone & Kozlowska) which is fixed.

Otherwise we make extensive use of refs in brackets... please let me know if not ok.

[oliviaguest]

@fredshone looks beautiful ☺️

[oliviaguest]

@editorialbot recommend-accept

[editorialbot]

Attempting dry run of processing paper acceptance...

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.5334/baw is OK
- 10.17226/22357 is OK
- 10.1080/12265934.2013.835118 is OK
- 10.14279/depositonce-9835 is OK
- 10.1016/j.procs.2021.03.089 is OK

MISSING DOIs

- No DOI given, and none found for title: Pandemic Activity Modifier: Intro
- No DOI given, and none found for title: European Transport Conference Papers 2022: Agent-b...
- No DOI given, and none found for title: ActivitySim: Large-Scale Agent-Based Activity Gene...

INVALID DOIs

- None

[editorialbot]

👋 @openjournals/sbcs-eics, this paper is ready to be accepted and published.

Check final proof 👉📄 Download article

If the paper PDF and the deposit XML files look good in openjournals/joss-papers#5265, then you can now move forward with accepting the submission by compiling again with the command @editorialbot accept

[oliviaguest]

@editorialbot accept

[editorialbot]

Doing it live! Attempting automated processing of paper acceptance...

[editorialbot]

Ensure proper citation by uploading a plain text CITATION.cff file to the default branch of your repository.

If using GitHub, a Cite this repository menu will appear in the About section, containing both APA and BibTeX formats. When exported to Zotero using a browser plugin, Zotero will automatically create an entry using the information contained in the .cff file.

You can copy the contents for your CITATION.cff file here:

CITATION.cff

cff-version: "1.2.0"
authors:
- family-names: Shone
  given-names: Fred
  orcid: "https://orcid.org/0009-0008-1079-0081"
- family-names: Chatziioannou
  given-names: Theodore
- family-names: Pickering
  given-names: Bryn
  orcid: "https://orcid.org/0000-0003-4044-6587"
- family-names: Kozlowska
  given-names: Kasia
- family-names: Fitzmaurice
  given-names: Michael
doi: 10.5281/zenodo.10948231
message: If you use this software, please cite our article in the
  Journal of Open Source Software.
preferred-citation:
  authors:
  - family-names: Shone
    given-names: Fred
    orcid: "https://orcid.org/0009-0008-1079-0081"
  - family-names: Chatziioannou
    given-names: Theodore
  - family-names: Pickering
    given-names: Bryn
    orcid: "https://orcid.org/0000-0003-4044-6587"
  - family-names: Kozlowska
    given-names: Kasia
  - family-names: Fitzmaurice
    given-names: Michael
  date-published: 2024-04-23
  doi: 10.21105/joss.06097
  issn: 2475-9066
  issue: 96
  journal: Journal of Open Source Software
  publisher:
    name: Open Journals
  start: 6097
  title: "PAM: Population Activity Modeller"
  type: article
  url: "https://joss.theoj.org/papers/10.21105/joss.06097"
  volume: 9
title: "PAM: Population Activity Modeller"

If the repository is not hosted on GitHub, a .cff file can still be uploaded to set your preferred citation. Users will be able to manually copy and paste the citation.

Find more information on .cff files here and here.

[editorialbot]

🐘🐘🐘 👉 Toot for this paper 👈 🐘🐘🐘

[editorialbot]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

0. Check final PDF and Crossref metadata that was deposited 👉 Creating pull request for 10.21105.joss.06097 joss-papers#5266
1. Wait five minutes, then verify that the paper DOI resolves https://doi.org/10.21105/joss.06097
2. If everything looks good, then close this review issue.
3. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? Notify your editorial technical team...

[fredshone]

thank you all 🥳

[oliviaguest]

Huge thanks to the reviewers and editor, @martinfleis, @jamesdamillington, @martibosch! ✨ JOSS appreciates your work and effort. ✨ Also, big congratulations to the authors @fredshone! 🥳 🍾

[editorialbot]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.06097/status.svg)](https://doi.org/10.21105/joss.06097)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.06097">
  <img src="https://joss.theoj.org/papers/10.21105/joss.06097/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.06097/status.svg
   :target: https://doi.org/10.21105/joss.06097

This is how it will look in your documentation:

We need your help!

The Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://reviewers.joss.theoj.org/join
• Making a small donation to support our running costs here: https://numfocus.org/donate-to-joss