openPMD: Collective Open & Group Based

[ax3l]

Re-introduce writing to one file to avoid syncs #270 / #273 / #271 (comment)

Write in parallel into one file, which we open collectively.
This has the nice side-effect, that later iteration flushes can be performed as non-collective writes.

To Do:

• decide on a test case that is checked in that we can do a runtime test on together: blowout_wake
• wait for CPU-side fixes in HiPACE (not needed for this test)
• test with CPU (1 rank)
• test with CPU (5 ranks)
• test with GPU (1 rank)
• test with GPU (5 ranks)

General:

• Small enough (< few 100s of lines), otherwise it should probably be split into smaller PRs
• Tested (describe the tests in the PR description)
• Runs on GPU (basic: the code compiles and run well with the new module)
• Contains an automated test (checksum and/or comparison with theory)
• Documented: all elements (classes and their members, functions, namespaces, etc.) are documented
• Constified (All that can be const is const)
• Code is clean (no unwanted comments, )
• Style and code conventions are respected at the bottom of https://github.com/Hi-PACE/hipace
• Proper label and GitHub project, if applicable

[SeverinDiederichs]

I would propose to use the blowout_wake example. It is super fast and produces a blowout wake to see if the fields are correct.
Calling it with mpiexec -n 5 ../../build/bin/hipace inputs_normalized max_step=20 hipace.verbose=1 enables to assess the performance of the pipeline. I would propose to use 5 ranks instead of 6, so we don't have to change the number of grid points in the input script.

No fixes on the CPU side are required (there are some corner cases as input parameters, which are failing, but they should not affect I/O at all).

@ax3l Do you agree on the approach?

[ax3l]

Sounds good - that works on my system. The only downside with the blowout_wake example is that the fields structure I see in output is so static for all steps >0 that it is easy to overlook if the data is not updated anymore.

What do I need to pass to the input file to compute with/without guards? Adding hipace.3d_on_host=1 hipace.slice_beam=1 for guards?

[ax3l]

Progress: the issue is not here but with openPMD-viewer. Will report back with findings.

[ax3l]

@SeverinDiederichs PR updated :)
I already tested the following:

• 3D and slice output
• visualization with openPMD-viewer dev using its openPMD-api backend

To do:

• ❌ visualization with openPMD-viewer dev using its h5py backend
• ❌ visualization with openPMD-viewer (current stable release) (tested by Severin)

openPMD-viewer bug report: openPMD/openPMD-viewer#299

[SeverinDiederichs]

Thanks, with the new openPMD-viewer I can verify that it indeed works as intended 🎉 . The test we agreed on shows differences between CPUs and GPUs, but this is the bug on CPUs we discussed earlier (I will link the issue after I have opened it). For stricter predictor corrector loop settings, both CPU and GPU produce the same results. I have tested slice I/O, 3D IO, all with 1 and 5 ranks on both CPU and GPU.