Slow down with repeated calls to ts.get_field() using io_reader #380
Comments
|
Thank you for the details! We chatted further on Slack and using HDF5 with group-based encoding should be way faster. I suspect that we accidentally close and open the same file all the time in openPMD-viewer - let's debug this! In the meantime, you might be faster if you read with openPMD-api for now until we fix this here: |
|
I have the same problem |
Does this still happen when you close Iterations with EDIT: Aah, just noticed that this is the openPMD-viewer. Yep, for that one we will probably need the functionality described above.. Will look at the rest of the issue tomorrow |
|
Here are the major ways how people iterate over data in openPMD-viewer: Time Series Explicit Accessfrom openpmd_viewer import OpenPMDTimeSeries
ts = OpenPMDTimeSeries('./example-2d/hdf5/')N_iterations = len(ts.iterations)
for i in N_iterations:
rho, info_rho = ts.get_field( iteration=ts.iteration[i], field='rho' )
# or more fields or particlesSame used with explicit iterations in the GUI https://openpmd-viewer.readthedocs.io/en/latest/tutorials/3_Introduction-to-the-GUI.html IterateSee #400 Note: after a first iterate, a user might want to do this again on the same time series. There are currently some limitations in Let's discuss if we can generalize this and potentially also add an iteration filter mode. For large processing tasks, this mode would allow for way more optimizations (e.g., visiting iterations once, freeing resources) than the explicit access in the example above. |
|
I created a small dataset to check things out. on frontier Total is 100G for 500steps. Using Remi's script , output reading file based mode: output with group based mode: So I don't observe the pattern Remi observed. Let me try BP4 next |
|
I think Remi used BP4 with file encoding. group encoding is stable. Processing file bp4-f/defaultBP4-node_diags/diag1/ Processing file bp4-g/defaultBP4-node_diags/diag1/ |
|
So for the time being, switch to BP5 would be the easiest workaround. Would be good to add read support of variable based. encoding in the openPMD-viewer. I think the performance should be similar to the group based. |
|
@guj I'm not sure if you're really measuring a slowdown and not just slight performance fluctuations:
@aveksler1 This is obviously difficult to debug from afar, but let's try. Please do only steps 1-3 of this so far since those might already confirm the cause that I think most likely:
There are two possible reasons that I currently see for this slowdown:
|
|
@franzpoeschel the plot time you have is for group based encodings.
|
Given that the openPMD-viewer is in this context a wrapper about the openPMD-api, some optimizations are probably possible today already, like closing an iteration when the next one is accessed. The downside is that an iteration once closed can currently not be reopened again, which can be internally worked-around by closing and reopening the Series (ideally with The access patterns themselves are probably fine, it's more a question of how they are implemented in the openPMD-viewer (or in future: in the openPMD-api). As discussed offline, one important next step for the openPMD-api is a better support for typical access patterns, with such workarounds ideally being moved into the openPMD-api. My idea is that depending on the access configuration, you get an iterator from |
|
While we generally think the solution will be to go to BP5, we currently see that file based encoding in BP5 is slower than BP4 in the benchmarks above. @guj and @pnorbert are investigating further. Issue #400 is a smaller reproducer of this issue, that contains HDF5 and ADIOS2 files. |
|
@aveksler1 if you still have the issue, can you profile your code like this? |
|
The general solution is to use https://warpx.readthedocs.io/en/latest/usage/parameters.html#diagnostics-and-output We plan for openpmd-api 0.16+ and according WarpX releases to go from BP4 to BP5 and to go from groupBased to variableBased encoding. |
|
@ax3l I'll run test cases with h5 and bp backends and different encodings, 'f' and 'g'. I'll compare the timings results from consecutive calls to |
|
I have found the main cause for this slowdown. In openPMD-api is currently not really well-optimized for Series with many iterations (my personal focus has been more on Series with few very big iterations), so researching this bug uncovered not only one single performance issue. The linked PR is hence not yet ready, as it contains other (less trivial) fixes, and additionally there are some weird findings that I still need to have a look at. Until that PR is merged, a workaround might be adding |
|
Thank you so much! I was able to avoid the problem by only using h5 output, but having the ability to rapidly post-process ADIOS2 files will be a great help when we start running simulations that benefit from ADIOS2 output. |
|
Performance analysis:
Action items:
|
|
Per #380 (comment) The specific issue is closed and will be released with openPMD-api 0.16.0. Generally, the overhead of large series with many iterations in open should be linear. Please continue to report issues if you see otherwise 🙏 |
Hi all,
I'm using
openPMD-viewerto loop through the outputs of a WarpX simulation and perform a calculation on fields at every iteration. I noticed that as I loop throughts.iterations, theget_fields()call takes longer and longer. I'm running this script on a local cluster, and thets.backendisopenpmd-api.The script where I am seeing slowdown looks like this. I am not able to recreate the problem using the openPMD example datasets. The dataset I am using (which I unfortunately cannot share) are 4000 iterations of E and B fields on a 72x72x144 grid. Every .bp directory is ~ 35 Mb, so around 140 Gb in total.
This outputs:
and this scales worse and worse as I increase the length of
iterations.I tracked down the offending function to be
get_datain io_reader/utilities, specifically theseries.flush()call repeated for every chunk. Timing it shows it takes up the majority of thets.get_fields()call.I don't see this behavior when the openPMD outputs are .h5 files (but still openpmd-api backend). Any help would be much appreciated!
The text was updated successfully, but these errors were encountered: