Batch running, local cache distributed to parallel processing workers, basic interactivity #17
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…t at least the hard / abstract part is done
…ight be easiest solution to multiproc / batch cache issue
|
Note that since I just created the |
… a single option for each other than the planting density which is what is causing all of the different sims:
…ranch protection with a basic integration test using pixi env
…now, within pyteest runner
…tnot. Definitely a cleaner way is to just use the docker image we build here locally, push it to registry, and re-use it - but that's more work ATM and we can address later
* Extract the pystac client to a cached property; it'll only be
initialized when it's accessed, and subsequent accesses return the
same object as the first access
* Extract cache existence check to own function for readability
However I still receive an error trying to start the app:
```
File "/home/robatt/Projects/dse/mesa_abm_poc/.pixi/envs/default/lib/python3.11/site-packages/stackstac/rio_reader.py", line 352, in _open
vrt = WarpedVRT(
^^^^^^^^^^
File "rasterio/_warp.pyx", line 1045, in rasterio._warp.WarpedVRTReaderBase.__init__
File "rasterio/crs.pyx", line 786, in rasterio.crs.CRS.from_user_input
File "rasterio/crs.pyx", line 596, in rasterio.crs.CRS.from_epsg
rasterio.errors.CRSError: The EPSG code is unknown. PROJ: internal_proj_create_from_database: /home/robatt/Projects/dse/mesa_abm_poc/.pixi/envs/default/share/proj/proj.db contains DATABASE.LAYOUT.VERSION.MINOR = 2 whereas a number >= 3 is expected. It comes from another PROJ installation.
```
It would be great if we could get the needed versions of mesa-geo up on conda-forge! The source of the problem seems to be mixing pip and conda deps. Rasterio >=1.4b1 is required by mesa-geo 0.9: https://github.com/projectmesa/mesa-geo/blob/bb1cfca2d799d4d5012d45578cc99dbb1c61af87/pyproject.toml#L46 And we specified a conda dependency of rasterio 1.3.*.
Use /tmp/ as cache base directory because we can expect it to exist on the system and be writeable as non-root. However, now we're stuck in a loop of writing to the same file over and over!
…ion model to get RasterLayer.to_file kicks because we don' have the cache, but that's the whole point. Really though we just need to use rio to export whatever this cache is meant to look like (or I guess fail silently or show a warning in Vegetation - since it won't be a problem until we step
…. solved the Vegetation instantiation problem by creating an method, which likely will have other knock on benefits down the line:
|
As expected this broke some functionality within this mega-branch - working now to resolve some of the mismatches between private attributes, env vars, etc etc. Hopefully not too painful 😅 |
…ut thought I might get lucky lol
… egg scenario here with the cache building logic, but this hacky solution works for now. Essentially we are triggering before we start the sim so that the solara Viz renders properly.
GondekNP
commented
Jan 14, 2025
| circlemarker={} | ||
| ) | ||
| tree_management.on_draw(model.add_agents_from_management_draw) | ||
| cache_manager.populate_elevation_cache_if_not_exists() |
There was a problem hiding this comment.
I had hoped this _on_start function could be used to pack in some behaviors that we want to make sure have happened before the simulation starts (pack exogenous variables, add agents to the grid, etc), such that we could instantiate a lazy Vegetation which could be used with CacheManager. To me this is one of those things we can tolerate for V0, but definitely qualifies as a code smell.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This branch is a work in progress, but main changes include
Batch jobs running via
vegetation/batch.py- which can in turn be invoked with the relevantlaunch.json. So far no configuration to actually capture the necessary outputs or do any parameter sweeps or anything (it is set up, but no relevant paramaters such as survival or management choices are included, just planting density which is only used interactively so doesn't actually affect the batch run)Fixes Batch run can't see local cache #16 by copying the local cache to the docker container after build - kind of clunky but works for now. We will certainly want a better way of caching / distribution down the line.
Adds basic
painting treeslogic - user can use the inlcuded polygon tool and when the polygon is closed, theon_drawevent will be captured and trees will be added to the simulation according tomanagement_planting_density. Logs will reflect the added trees as well.Known issues-
Some mismatch between the lat/lon getting converted to proper indices within mesa - not a huge deal but need to look back on my convo with mesa devs (Invalid pos/index within RasterLayer (geographic spatial location not exposed to GeoSpace / RasterLayer)? projectmesa/mesa-geo#267) to see what's going wrong here
Autoreload in Solara is acting up - the first render does not work, only subsequent ones do