BP5 Shape Resize without Empty Put

[ax3l]

We discussed on January 30th, 2023 in our ADIOS2-PIC development meeting the following idea to improve meta-data for particle tracking in BP5.

In BP5, global info for all blocks is stored in the meta-data once, at EndStep.

Currently, this requires a Put to be called, which is hard for MPI-parallel API contracts to convey, because Put is a non-collective API call. SetShape on the other hand is clearly collective.

What could be done is use the info from SetShape in EndStep, instead of needing an empty Put from MPI ranks that do not add to the newly resized variable.

Implementation:

• in EndStep, go through all variables and take their shape.

Downside: none; seems it cannot be fixed in BP4 the same way.

Pro: SetShape is collective and Put a truly independent and side-effect free in API contracts.

Related issue:

• storeChunk with zero extent openPMD/openPMD-api#1147
• really unpleasant work-around for the current BP4 and BP5 logic when dealing with particles: ADIOS2: Fix BTD Particle Resize w/ Empty Ranks ECP-WarpX/WarpX#3657

[ax3l]

attn @pnorbert @guj @franzpoeschel

[eisenhauer]

You might think that SetShape would be collective because it seems like something that the ranks would have to agree upon, but nothing actually enforces that in BP5. As you've observed, we only snag Shape on Put(), but we could, at least with the current implementation, stop doing that and instead walk the set of Put() variables in EndStep and put the final Shape in the metadata. But in the context of that let me ask a question. In BP5 we implicitly rely upon the idea that the writer ranks have agreed upon a Shape. Unlike BP4, we don't "merge" metadata from different ranks at any point, instead readers rely upon the Shape value that we get from writer rank 0 (or from the lowest rank that actually had a Put() for that variable). Does that cause you any problem? (I'd worry about doing something more complex than this on the reader size, like requiring BP5 to scan all the writer ranks to pick the biggest shape that it finds, because that would add overhead to all read-side metadata processing to accommodate the presumably rare circumstance when the application ranks don't agree on the Shape.)

[ax3l]

Thanks for the feedback & questions, Greg!

As you've observed, we only snag Shape on Put(), but we could, at least with the current implementation, stop doing that and instead walk the set of Put() variables in EndStep and put the final Shape in the metadata.

yes, that would be our preferred solution.

In BP5 we implicitly rely upon the idea that the writer ranks have agreed upon a Shape. Unlike BP4, we don't "merge" metadata from different ranks at any point, instead readers rely upon the Shape value that we get from writer rank 0 (or from the lowest rank that actually had a Put() for that variable). Does that cause you any problem?

No, that is fine. We know the new (resized) global shape on all ranks, even if a rank does not contribute new particles during a resize with a Put.

To be more precise, we collect this costly All2All info just for I/O #3452, but yes, we know it and can guarantee all ranks tell you the same info.

[ax3l]

I have a question for you, since we often default to BP4 in our production workflows.

Unlike BP4, we don't "merge" metadata from different ranks at any point

Do you think there is a way to fix this also in BP4? The need for an empty Put from every rank for every variable is really unpleasant to code and keep book of. This is a super common I/O pattern for mesh-refinement simulations, particle tracking, etc. even if the concept of resizing a global shape is relatively new in ADIOS.

Unrelated but just a note: HDF5 does also not need such extra empty-write calls to update the global shape. It just needs the guarantee that the shape meta-data is updated collectively.

[eisenhauer]

Maybe theoretically possible, but it might well be of huge practical difficulty. I'm less familiar with the details of BP4 metadata processing. I believe that in EndStep it's all encoded, pulled to rank 0, decoded and then re-encoded as a more unified thing. (BP5 does the first two things, but not the decode/re-encode.) The last re-encode might be where you could put something like that in BP4, but it might not be reasonable depending upon the structure of the code, what data structures it builds up during the process, etc.

I'll see about making these changes in BP5 and putting together a test for updating Shape during a timestep...

[eisenhauer]

Hey Axel! Just curious, is this functionality was still necessary (or at least useful), or does the implementation of Joined Arrays covers the use case?