[WIP] Convert openPMD frontend to use openPMD-api

[henrynjones]

This is a work in progress to change the openPMD frontend to be able to read in all openPMD markup files regardless of backend and add mesh refinement capabilities.

PR Summary

This work-in-progress PR addresses issue #4757 by using the backend-agnostic openPMD-api as the IO module instead of the previous h5py.

TODO

• Add sorted particles to the particle grid in which they are located, for the short term this will only work for ADIOS2, file-based iteration encoded datasets with the particlePatches.

• Ensure that HDF5 behavior is preserved if not improved.

  • For HDF5 files, record_component.available_chunks() only contains one chunk and all levels cover the entire domain. We might have to trim/mask higher level data so that only level 0 data exists on the entire domain, and each subsequent level covers only the refined region.

• Still haven't looked at string handing for file path patterns. Would be great to load a dataset series from file-based iteration-encoded diagnostics. Right now yt uses [0-9] or dd???/dd??? or diag_name_??? while openpmd-api uses %T or %06T for example.

• Testing has mostly been done using 2d WarpX simulations, where the axes are 'x' and 'z' which causes problems with axis labeling and the orientation of plots, and thus the annotation of particles onto SlicePlots. Work is being done to use the openpmd axis_labels (and later fortran capability with data_order?) to correctly align axes as written.

• Eventually get to doing yt tests

PR Checklist

• New features are documented, with docstrings and narrative docs
  • Modern mesh refinement tools similar to the boxlib frontend for ADIOS2 files. Right now this only seems to work with file-based iteration encoding schemes.
• Adds a test for any bugs fixed. Adds tests for new features.

[chrishavlin]

in addition to adding openPMD-api to on_demand_imports you'll want to add it to the pyproject.yaml file, which has optional dependencies organized by frontend, here's the current open-pmd entry:

yt/pyproject.toml

Line 118 in 303991e

open-pmd = ["yt[HDF5]"]

[henrynjones]

Hello, I have a lot to learn with git but basically in trying to squash my commits in a rebase I have squashed all commits into one. Any recommendations for how I can just extract the differences in main into a single new commit to place on top of the yt history? Thanks

A sample notebook and more detailed documentation with my changes and current issues are in the works.

[neutrinoceros]

Any recommendations for how I can just extract the differences in main into a single new commit to place on top of the yt history? Thanks

I think at this point we need to nuke your branch and start afresh. Here's how you can salvage the diff.

# fetch the latest state of the main branch
git remote add upstream https://github.com/yt-project/yt
git checkout main
git pull upstream main

# create a savefile from your current branch
git checkout openmd_update
pre-commit run --all-files # auto clean trailing whitespaces and other lints
git diff main > openmd_update.patch

# switch to a new branch from main and apply the patch there
git switch openmd_update_2 main
git apply openmd_update.patch
git commit -am "<insert your commit message here>" --no-verify
git push --set-upstream origin openmd_update_2

note that you'll need pre-commit installed (I found that this process didn't work without it).