Feature/asv by fluidnumerics-joe · Pull Request #25 · Parcels-code/parcels-benchmarks

fluidnumerics-joe · 2025-12-10T22:00:28Z

This PR transitions all benchmarking to use ASV.

Currently asv refuses to run with multiple parameters

fluidnumerics-joe · 2025-12-10T22:04:48Z

@VeckoTheGecko - you might give this a try. On this branch, do

pixi install
pixi shell

Once in the pixi shell, try,

asv run --python=same --quick --show-stderr --dry-run --verbose

At the moment, asv is not picking up any benchmarks and its unclear why... output shown below

$ asv run --python=same --quick --show-stderr --dry-run --verbose
· Running '/home/joe/Projects/Geomar-Utrecht/parcels-benchmarks/.pixi/envs/default/bin/python -c import sys; print(str(sys.version_info[0]) + "." + str(sys.version_info[1]))'
  OUTPUT -------->
  3.12
· Discovering benchmarks
·· Running '/home/joe/Projects/Geomar-Utrecht/parcels-benchmarks/.pixi/envs/default/lib/python3.12/site-packages/asv/benchmark.py discover /home/joe/Projects/Geomar-Utrecht/parcels-benchmarks/benchmarks /tmp/tmphyvjz_3_/result.json' in existing-py_home_joe_Projects_Geomar-Utrecht_parcels-benchmarks_.pixi_envs_default_bin_python
·· Running '/home/joe/Projects/Geomar-Utrecht/parcels-benchmarks/.pixi/envs/default/bin/python /home/joe/Projects/Geomar-Utrecht/parcels-benchmarks/.pixi/envs/default/lib/python3.12/site-packages/asv/benchmark.py discover /home/joe/Projects/Geomar-Utrecht/parcels-benchmarks/benchmarks /tmp/tmphyvjz_3_/result.json'
   OUTPUT -------->
   /home/joe/Projects/Geomar-Utrecht/parcels-benchmarks/benchmarks/moi_curvilinear.py:8: UserWarning: This is an alpha version of Parcels v4. The API is not stable and may change without deprecation warnings.
     import parcels
· No benchmarks selected

fluidnumerics-joe · 2025-12-10T22:05:19Z

Right now this is set up to use all packages (including parcels) as defined by the pixi environment.

…problem" This reverts commit 0d46b63.

fluidnumerics-joe · 2025-12-11T01:28:19Z

Looks like benchmarks need to be prefixed with specific names, e.g. time_ or peakmem_ so that asv picks them up.. See https://asv.readthedocs.io/en/stable/benchmarks.html

asv.conf.json

Pixi environment now only provides the necessary packages for the benchmarking environment. asv will use rattler to install parcels and its dependencies in another environment

…hmarks The changes provided here allow for the parcels_bencharks/benchmark_setup.py module to be found within asv benchmarks. This makes it easy to handle downloading the necessary datasets for benchmarks. I've added a helper function in the moi curvilinear benchmarks to load the xarray dataset; storing the dataset as an object attribute can cause contamination between benchmarks that I want to avoid. Each benchmark now loads a fresh dataset from disk at the beginning by calling _load_ds(...)

…hmarks

willirath · 2025-12-12T14:38:02Z

@willirath - I'm honestly inclined to remove the cli and just keep the data downloading in the 'setup' of each benchmark. The gist was to provide a means for folks to pre-download the data. While yes, there's no direct enforcement of the --data-home flag and the data_home in the benchmarks, if someone is using something other than the default, they'll need to modify the benchmark.

This is really a half baked idea that requires folks know what they're doing here at the moment.

What about removing the CLI but adding a DATA_DIR env var for override? Without this, I have to clean out my $HOME/.cache dir each time I run these on Levante benchmarks because DKRZ.de is really strict about $HOME quota.

fluidnumerics-joe · 2025-12-12T16:01:55Z

@willirath - makes sense. Let me see what I could do to tighten up the connection here wrt to the data home

Co-authored-by: Willi Rath <willirath@users.noreply.github.com>

Until we determine how we want to manage centrally storing benchmark results, we'll keep these out of the repository for now

fluidnumerics-joe · 2025-12-12T20:22:04Z

@VeckoTheGecko - I've removed the .asv/ subdirectory and added .asv/ to .gitignore. I figured it'd be best to discuss in a separate issue how we want to centrally manage benchmarks submitted by multiple users. IMO, having contributions from the developers and any interested members of the Parcels community would be awesome; this way we can concretely see the variation in performance across Parcels users' systems.

fluidnumerics-joe · 2025-12-12T20:23:47Z

@willirath - I think I've addressed all of your comments. You can now use the PARCELS_DATADIR environment variable to control where the local data cache is stored. By default, it points to whatever pooch.os_cache returns with. The CLI is removed and the benchmark_setup.py script simply can be used to pre-download all data before running benchmarks, if desired.

parcels_benchmarks/benchmark_setup.py

willirath

Two minor things: leftover argparse logic in main block and missing Path on env var.

parcels_benchmarks/benchmark_setup.py

Co-authored-by: Willi Rath <willirath@users.noreply.github.com>

parcels_benchmarks/benchmark_setup.py

Co-authored-by: Willi Rath <willirath@users.noreply.github.com>

willirath · 2025-12-14T13:05:05Z

Just tested, both, with / without overriding the cache dir on our Uni Kiel HPC and on my Macbook. I think it's good to be merged.

erikvansebille

Looks good! Nice push forward. And installation was a breeze 👍

A few comments below

erikvansebille · 2025-12-15T15:50:21Z

benchmarks/moi_curvilinear.py

+
+        pset.execute(parcels.kernels.AdvectionEE, runtime=runtime, dt=dt, verbose_progress=False)
+
+    def peakmem_pset_execute_3d(self,interpolator,chunk,npart):


The body of this function appears identical(?) to the body of the time_pset_execute_3d()? Can we reduce code duplication by organising this more smartly? Why do we need two functions?

They are identical, but this is the way ASV works. If we want a benchmark for measuring peak memory the function has to start with peakmem_. If we want runtime, it has to start with time_

OK, but can't we then have these two functions that then call another function _execute() or so, so that the execute remains the same. Also important to avoid one benchmark changing but the other not?

erikvansebille · 2025-12-15T15:50:42Z

parcels_benchmarks/__init__.py

@@ -0,0 +1 @@
+# parcels_benchmarks/benchmark_utils


Intentionally commented out?

parcels_benchmarks/benchmark_setup.py

erikvansebille · 2025-12-15T15:53:22Z

.gitignore

+__pycache__
+build/
+parcels/
+.asv/


I understand @VeckoTheGecko comment to ignore the individual run outputs, but how/where are they stored now then?

When you run, the results are stored under .asv/ . I commented in #24 to discuss what we should retain here in version control as a follow up :)

README.md

erikvansebille · 2025-12-15T15:57:00Z

Oops, I now only realise the PR was merged already. Well; perhaps some of my comments are useful for other rounds of enhancements?

fluidnumerics-joe · 2025-12-15T16:05:02Z

Oops, I now only realise the PR was merged already. Well; perhaps some of my comments are useful for other rounds of enhancements?

Yes indeed

fluidnumerics-joe added 3 commits December 10, 2025 13:22

Start cleanout of old scripts

2675407

Add cli to setup tools; fix registry loading

dbb1d16

Reorganize moi_curvilinear benchmarks for use with asv

4355b74

Currently asv refuses to run with multiple parameters

fluidnumerics-joe requested a review from VeckoTheGecko December 10, 2025 22:00

fluidnumerics-joe added 4 commits December 10, 2025 17:09

Remove parameters to try an isolate "No benchmarks selected" problem

0d46b63

Fix branches definition

dbec40b

Revert "Remove parameters to try an isolate "No benchmarks selected" …

dcf5304

…problem" This reverts commit 0d46b63.

Prefix benchmarks with time_ and mem_

e5e71e7

fluidnumerics-joe added 3 commits December 10, 2025 20:33

Fix runtime errors with moi benchmarks

73b2947

fix reference to self.interp_method

d87ea4a

Fix range of time loop for load data benchmarks

cf8a2f5

VeckoTheGecko reviewed Dec 11, 2025

View reviewed changes

asv.conf.json Outdated Show resolved Hide resolved

fluidnumerics-joe added 14 commits December 11, 2025 06:07

Change to rattler asv environment

fdb5963

Pixi environment now only provides the necessary packages for the benchmarking environment. asv will use rattler to install parcels and its dependencies in another environment

Add conda channels for rattler

8eb8b4b

Update benchmarking steps

19e8f5b

Only load two time and depth levels for io benchmark

efbd0ef

Add fesom2 benchmarks

1845fb5

Trim out unused modules

0afc5a3

Add __init__.py for local utils

433658e

Clean out old benchmarks file

0d08f9c

Set mesh type in UxGrid creation

0f97130

Add more instructions for contributing benchmark data and adding benc…

b59960a

…hmarks

Fix data array names in fesom io benchmark

9d972ff

Add steps for visualizing the data

be5616e

Add asv results

d9f6310

fluidnumerics-joe changed the title ~~[WIP] Feature/asv~~ Feature/asv Dec 11, 2025

fluidnumerics-joe and others added 9 commits December 12, 2025 13:33

Remove parcels._v3to4 module import

fd4c5bf

Fix docstring for download_example_dataset

fa7467a

Co-authored-by: Willi Rath <willirath@users.noreply.github.com>

Fix docstring in download_datasets

e861018

Co-authored-by: Willi Rath <willirath@users.noreply.github.com>

Fix list_dataset - DATA_FILES only has a single zip file name

282b5cc

Co-authored-by: Willi Rath <willirath@users.noreply.github.com>

Fix help string for --data-home flag

9341080

Co-authored-by: Willi Rath <willirath@users.noreply.github.com>

Remove retrieve cli

b8a82aa

Co-authored-by: Willi Rath <willirath@users.noreply.github.com>

Remove retrieve data directory call

a01db1b

Co-authored-by: Willi Rath <willirath@users.noreply.github.com>

Remove CLI from setup; enforce datahome via PARCELS_DATADIR env variable

d3e7426

Remove .asv subdirectory

845ba0a

Until we determine how we want to manage centrally storing benchmark results, we'll keep these out of the repository for now

fluidnumerics-joe requested a review from willirath December 12, 2025 20:24

willirath reviewed Dec 13, 2025

View reviewed changes

parcels_benchmarks/benchmark_setup.py Outdated Show resolved Hide resolved

willirath requested changes Dec 13, 2025

View reviewed changes

parcels_benchmarks/benchmark_setup.py Outdated Show resolved Hide resolved

fluidnumerics-joe and others added 2 commits December 13, 2025 08:45

Remove stale call to cli in main()

626f252

Co-authored-by: Willi Rath <willirath@users.noreply.github.com>

Set PARCELS_DATADIR as Path type

c911aba

Co-authored-by: Willi Rath <willirath@users.noreply.github.com>

willirath approved these changes Dec 14, 2025

View reviewed changes

willirath reviewed Dec 14, 2025

View reviewed changes

parcels_benchmarks/benchmark_setup.py Outdated Show resolved Hide resolved

Update parcels_benchmarks/benchmark_setup.py

f5de95b

Co-authored-by: Willi Rath <willirath@users.noreply.github.com>

fluidnumerics-joe merged commit 905c8cb into main Dec 15, 2025

fluidnumerics-joe deleted the feature/asv branch December 15, 2025 15:54

erikvansebille reviewed Dec 15, 2025

View reviewed changes

fluidnumerics-joe mentioned this pull request Dec 15, 2025

ASV #24

Open

erikvansebille mentioned this pull request Dec 15, 2025

Provide info how the known_hash can be discovered? #26

Closed

fluidnumerics-joe mentioned this pull request Dec 15, 2025

Define single function for particle forward stepping #27

Merged


		pset.execute(parcels.kernels.AdvectionEE, runtime=runtime, dt=dt, verbose_progress=False)

		def peakmem_pset_execute_3d(self,interpolator,chunk,npart):

Conversation

fluidnumerics-joe commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fluidnumerics-joe commented Dec 10, 2025

Uh oh!

fluidnumerics-joe commented Dec 10, 2025

Uh oh!

fluidnumerics-joe commented Dec 11, 2025

Uh oh!

Uh oh!

willirath commented Dec 12, 2025

Uh oh!

fluidnumerics-joe commented Dec 12, 2025

Uh oh!

fluidnumerics-joe commented Dec 12, 2025

Uh oh!

fluidnumerics-joe commented Dec 12, 2025

Uh oh!

Uh oh!

willirath left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

willirath commented Dec 14, 2025

Uh oh!

erikvansebille left a comment

Choose a reason for hiding this comment

Uh oh!

erikvansebille Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

fluidnumerics-joe Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

erikvansebille Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

erikvansebille Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

erikvansebille Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

fluidnumerics-joe Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

erikvansebille commented Dec 15, 2025

Uh oh!

fluidnumerics-joe commented Dec 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fluidnumerics-joe commented Dec 10, 2025 •

edited

Loading