[REVIEW]: Mesa 3: Agent-based modelling with Python in 2024

[editorialbot]

Submitting author: @EwoutH (Ewout ter Hoeven)
Repository: https://github.com/projectmesa/mesa
Branch with paper.md (empty if default branch): paper_2024
Version: v3.1
Editor: @gkthiruvathukal
Reviewers: @martibosch, @jofmi
Archive: Pending

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/0c0e31521267ed506b9f352cba409559"><img src="https://joss.theoj.org/papers/0c0e31521267ed506b9f352cba409559/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/0c0e31521267ed506b9f352cba409559/status.svg)](https://joss.theoj.org/papers/0c0e31521267ed506b9f352cba409559)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@martibosch & @jofmi, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review.
First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @gkthiruvathukal know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @jofmi

[editorialbot]

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.98  T=0.15 s (1027.5 files/s, 178491.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          86           3392           3808           9905
Markdown                        35           1298             26           3429
Jupyter Notebook                 5              0           2145            752
YAML                            12             19             39            356
DOS Batch                        1             29              1            212
TeX                              2             14              0            157
make                             1             28              6            143
TOML                             1              9             17            127
Text                             3             16              0             69
JSON                             1              0              0             12
Dockerfile                       1              9             19              9
Bourne Shell                     1              1              0              7
INI                              1              0              0              2
-------------------------------------------------------------------------------
SUM:                           150           4815           6061          15180
-------------------------------------------------------------------------------

Commit count by author:

   420	rht
   301	Ewout ter Hoeven
   284	jackiekazil
   120	David Masad
   115	Jan Kwakkel
   101	Corvince
    76	Jackie Kazil
    71	tpike3
    58	Tortar
    43	pre-commit-ci[bot]
    42	Taylor Mutch
    24	cauemello
    23	Dustin J. Mitchell
    22	Jeetu
    19	corvince
    17	ReblochonMasque
    16	Tom Pike
    13	Drewrey Lupton
    13	Jon Schiefelbein
    13	Wang Boyu
    12	SebastianoF
    12	Thomas Johnson
    11	dependabot[bot]
    10	Animesh Rawat
    10	Matt Davis
     8	James Arruda
     8	Nathan Miller
     8	Stephen Mubita
     7	Ankit Kumar
     7	Joe Dight
     7	gcallah
     7	ihopethiswillfi
     7	rlskoeser
     6	Achal Jain
     6	Heß
     6	Jessica Garson
     6	Spartan-71
     6	Steven MacLeod
     6	deepsource-autofix[bot]
     5	Dan Cunningham
     5	Daniel Chen
     5	Keith Richards
     5	Sascha Holzhauer
     5	Yannick Jadoul
     5	devforfu
     5	nissu99
     4	Ash Hoover
     4	Brian Russo
     4	Colin Maxwell
     4	DanielWeitzenfeld
     4	James Hovet
     4	Jangsea Park
     4	Max Orhai
     4	Nathan Verzemnieks
     4	Raúl Reguillo Carmona
     4	jess010
     4	jkerfs
     4	jonahaag
     3	AdamZh0u
     3	Benjamin Crane
     3	Bill Tucker
     3	Glenn Lehman
     3	ItsQuinnMoore
     3	Jatin Khilnani
     3	Jeremy Silver
     3	Keelan
     3	Martin Breuss
     3	Paul Gowder
     3	Yigit Guneli
     3	trickroll
     3	uec561
     2	Adam Zh0u
     2	Casper Smet
     2	Catherine Devlin
     2	Daniel Furman
     2	David
     2	David Balash
     2	Florian Kohrt
     2	Gonçalo Pascoal
     2	Jeremy Foote
     2	Luke Meyers
     2	Mat-pa
     2	Rebecca Sutton Koeser
     2	Ron Dotsch
     2	Sahil Chhoker
     2	Simon Podhajsky
     2	Victor Norman
     2	Vincent Hess
     2	federico
     2	houssam7737
     1	A Lee
     1	Abdallah Akrab
     1	Aditya Tiwari
     1	Alex
     1	Allen Lee
     1	Bubu
     1	Charlie Roe
     1	Christopher Akiki
     1	DarshPareek
     1	Deborah Duong
     1	Dev Test
     1	Edward Betts
     1	Eleonore9
     1	EwoutH
     1	Francis dela Cruz
     1	Giacomo Ravaioli
     1	Grace Z
     1	Houssam Kherraz
     1	Ikko Eltociear Ashimine
     1	J. Fernando Sánchez
     1	Jack at Omen Apps
     1	Jacob Hwang
     1	James Paterson
     1	Jared Norman
     1	Jisk Attema
     1	JoeHelbing
     1	Josiah Vinson
     1	LizaRoelofsen
     1	Luca Verginer
     1	LuciaBaldassini
     1	Maarten van der Sande
     1	Maarten-vd-Sande
     1	Majd Al-Shihabi
     1	Matt Wiens
     1	Megan Sosey
     1	Nick Aldershof
     1	Nouamane Tazi
     1	Oliver Katz
     1	Oren Bochman
     1	Paolo Gervasoni Vila
     1	Rebecca Davidsson
     1	Robert Hopkins
     1	Sahil Singh
     1	Sangamesh Itagi
     1	Sebastian Ortan
     1	Shengyu Huang
     1	Sherman
     1	Shubhanshu Mishra
     1	Sophia Wassermann
     1	Spencer Williams
     1	Steffen Rehberg
     1	Stephen Mann
     1	Taylor Mitchell
     1	Valentina Kiryushkina
     1	Vítor Fróis
     1	Wouter
     1	WouterVrielink
     1	Yasin Amini
     1	alba ramos
     1	clumdee
     1	electric-souperman
     1	gary
     1	gboeing
     1	jurreaserna
     1	kennethells
     1	logeaux
     1	maskarb
     1	nvrzm
     1	puer-robustus
     1	sahusiddharth
     1	schand5
     1	subhamonsey
     1	tcdejong
     1	vhess
     1	werwty

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

✅ OK DOIs

- 10.1063/1.1571309 is OK
- 10.2307/134812 is OK
- 10.25080/Majora-7b98e3ed-009 is OK
- 10.1007/978-3-030-61255-9_30 is OK
- 10.1145/3557989.3566157 is OK
- 10.1016/j.jclepro.2020.120296 is OK
- 10.1016/j.trd.2020.102455 is OK
- 10.1016/j.enbuild.2020.109763 is OK
- 10.1016/j.enpol.2017.08.024 is OK
- 10.1016/j.eswa.2022.116604 is OK
- 10.1016/j.elerap.2022.101195 is OK
- 10.1073/pnas.2215675120 is OK

🟡 SKIP DOIs

- No DOI given, and none found for title: Growing Artificial Societies: Social Science from ...
- No DOI given, and none found for title: Agent-based computational models and generative so...

❌ MISSING DOIs

- None

❌ INVALID DOIs

- None

[editorialbot]

Paper file info:

📄 Wordcount for paper.md is 1660

✅ The paper includes a Statement of need section

[editorialbot]

License info:

✅ License found: Apache License 2.0 (Valid open source OSI approved license)

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[jofmi]

Review checklist for @jofmi

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://github.com/projectmesa/mesa?
• License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@EwoutH) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1. Contribute to the software 2. Report issues or problems with the software 3. Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[EwoutH]

@martibosch @jofmi thanks for picking up the review! Some resources:

• The paper branch
• Our tracking issue: Tracking issue: JOSS paper projectmesa/mesa#2559

If you need anything else, let me know!

[EwoutH]

@martibosch @jofmi could you give us an estimated timeline on your reviews? In January I have a lot of time to discuss and implement feedback, February will be a bit more sparse.

[jofmi]

@EwoutH I will try to manage next week, latest first week of February! I have already spent some time with the paper, but still have to test the MESA code in practice :)

[EwoutH]

Awesome! If you very quickly want to run some examples, we also have an online demo:

• https://py.cafe/app/EwoutH/mesa-solara-basic-examples

[jackiekazil]

@jofmi
There was a minor change to the research mentioned in the paper.
No other changes to the paper are expected. It seems to me that there is some sort of publishing automation that JOSS has that needs to be triggered. Can you trigger it?

[EwoutH]

@editorialbot generate pdf

[EwoutH]

@martibosch I’m also curious what your estimated timeline is for a review.

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[gkthiruvathukal]

Please let me know if you need anything from me as editor.

[EwoutH]

Hi @martibosch, I hope you’re doing well!

Since we haven’t heard from you in almost 2 weeks, are you still interested in reviewing our Mesa paper? It’s okay if it takes some time, but it would be nice to hear something. And if not we can start the search for another reviewer.

[EwoutH]

@gkthiruvathukal maybe we need to start looking for an alternate reviewer in case Martí doesn't get back to us.

[martibosch]

Hello! sorry for the delay in my response. I plan to go through the library in mid-february, would that work for you?

[EwoutH]

Thanks for getting back!

I’m starting a new job around then, so I was hoping on a bit earlier. But since Joël will likely review this week or next, we can already work on that feedback.

Keep us in the loop!

[jackiekazil]

@EwoutH - I can pick up / support if needed.

[gkthiruvathukal]

Let me know, @EwoutH. It is easy to add another reviewer, e.g. @jackiekazil. As long as we have two complete reviews, we're fine. It seems like we still have @martibosch.

[jackiekazil]

I am an author on this paper. My offer was to support @EwoutH. I can't be a reviewer. :-)

[gkthiruvathukal]

Sorry, @jackiekazil, for the confusion! I have been dealing with a bit of a crisis at the office today. I propose that we proceed with the reviewers and their availability and revisit after a week or two.

[jofmi]

Dear All, I have now finished my review, thanks for the patience!

First of all, congratulations on this new version of MESA! The package has improved significantly since I last worked with it many years ago. Many of the challenges I encountered back then have been addressed. In particular, I think it was a good decision to deprecate the scheduler and the new visualization module is very impressive.

For transparency, I have previously also developed a Python-based ABM package (AgentPy). Given the advancements in this new version of MESA, I will also update the AgentPy documentation to reflect that the package is now largely deprecated in favor of MESA.

Below, you will find my comments presented in a checklist format. This will allow us to systematically address and mark them as resolved.

Major comments

• Experimental features: cell spaces and event-based scheduling (lines 76 & 86)
  • I believe these features should be completed before publication. Publishing a work in progress is not ideal, particularly since this is not the first paper on this package and these features appear to be close to a stable release.
  • Additionally, when I attempted to test these new features - which I find to be very interesting innovations - I found the online documentation lacking sufficient information.
  • If completing these features before publication is not feasible, I recommend moving them to a dedicated section such as Future Plans or Next Steps.

Comments on the documentation

• On the index page, I would add a sentence explaining what the "rec" and "all" extensions are needed for respectively (e.g. the visualization package?).
• I would suggest to put the "Overview of the MESA library" section within "Getting started" as a seperate page, as it is a handy reference that is more accessible than the API documentation, also after getting started.
• In addition, it would be great for the overview to cover all features of MESA. Compared to the paper, spaces and time advancement are missing here.
• In the introductory tutorial, it is a bit confusing what is supposed to go in the jupyter or the python file. For simplification of the tutorial, I would recommend for the whole tutorial to be within the jupyter file. If you want to keep the recommendation to split up files, I would add it as a tipp in "Best practices".
• For the setup, using code cells with terminal commands "!pip install --upgrade mesa[rec]" (note the exclamation mark at the start) enables installation within the jupyter file, removing the need for special instructions for google collab.
• Please put line breaks into the comments that are longer than 79 characters, allowing the documentation do be read on half-screen.
• Some sentences do not start capitalized.
• In the visualization tutorial, I would put the whole code of "MoneyModel.py" within the jupyter notebook, removing the need to search through the first tutorial for a file that is not really described.
• Maybe mention that the visualization needs the "rec" dependencies in the installation?
• I find the "Choose version" button at the top confusing, as it does not display the currently chosen version and there is already an indication on the bottom right.

Comments on the code

• The quality of the code and tests seem very good!
• Typing
  • One thing that could be positively highlighted is that MESA is using type hints. However, some additions would be needed for it to be able to run in a strongly typed environment.
  • In the introductory tutorial, chapter "Agents Exchange"/"Running your first model", I get a type error for "other_agent.wealth"/"a.wealth". It seems that model.agents does not return the correct agent type.
  • Same with using self.model within agent, the Agent does not know about the model types. (e.g. model.grid)

Comments on the paper

• l16 The Statement of Need could be more clearly structured. Currently, it reads more like a historical recap. I recommend organizing it along these lines: (1) Why such a package is needed, (2) The current state of the field, and (3) What MESA contributes — focusing on its present state rather than its 2014 origins. Additionally, please refer to the Statement of Need and State of the Field requirements outlined in the JOSS checklist above to ensure alignment with the journal's expectations.
• l41 For consistency with the other sections, please add a code example of instantiating a model and passing a seed
• l46 Similarly, please add a code example of instantiating and subclassing an agent
• l80 & l86 Maybe call these two 'modes' "discrete time" and "continuous time", if I understood it correctly? I think that would make it easier to understand.
• l109 I would suggest to move the "Applications" section to the start of the paper
• l134 Acknowledgements are usually placed after the conclusions

Smaller comments on the paper

• l15 Change to 2025
• l17 Not all ABMs represent artificial societies, here it is an implied synonym
• l20 The two references need to be placed into the same bracket
• l28 I think it should say here: "an" improved visualization framework
• l34 What does pure python mean? Does it not also use C, JS/React, etc.?
• l35 Markdown mistake in list formatting
• l43 Word repetition: "instantiates"
• l46 Please add a sentence describing agent types, to understand what the types mean later at "Data collection"
• l62-64 Please shortly explain these terms or add a reference
• l94 Markdown mistake, close bracket for link
• l96 Please add a link or reference regarding the Solara package
• l97 Please use a code example and screenshot that correspond to each other, demonstrating what kind of code creates what kind of visualization.
• l98 Maybe add a sentence how this visualization module aligns or differs with the established standard of NetLogo.
• l126 Markdown mistake in list formatting
• l133 Maybe add a sentence on which range of topics are covered by these community extensions and add a link or reference to a list or overview of these extensions and tutorials same as with the example models
• l143 Maybe say "want" instead of "need"?

[jackiekazil]

@jofmi Thank you for the review and the checklist! It is helpful!
We will make some updates and get back!

[EwoutH]

@jofmi thanks a lot for your detailed and thorough review! It’s really nice to have a fresh sets of eyes on it.

We’re tracking in:

• Requested Edits from JOSS projectmesa/mesa#2666

[jackiekazil]

@jofmi RE: l62-64 Please shortly explain these terms or add a reference
^^ Do you want all terms explained? Or is there something in particular that needs clarification? Asking for clarification to make sure we hit the mark.

[jofmi]

@jackiekazil particularly "voronoiMesh for irregular tessellations", but for new readers it might also be good to have a reference or a sentence regarding the moore and vonNeumann difference.

but i won't insist on this point, just thought it could be useful :)