Do not shrink serialization buffer upon EndStep – Fix Performance Bug #2516
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The BP4 serialization engine maintains the same serialization buffer
format::BP4Serializer
across steps. This allows to avoid heavyweight allocation operations after a certain "warmup" phase. Methods such asformat::BPBase::ResizeBuffer
support this idea by taking care not to shrink the buffer.The method
format::BP4Serializer::SerializeDataBuffer
(called uponEngine::EndStep
) does, however, pay no such attention.This has noticeable consequences in writing to ADIOS2 via openPMD in PIConGPU. (Possibly related to the fact that the openPMD plugin in PIConGPU will repeatedly call
Engine::PerformPuts
, leading to a low buffer "fill status" uponEngine::EndStep
, resulting in an actual shrinking of the buffer). While the buffer is never actually reallocated (which would happen only uponstd::vector::shrink_to_fit
), resizing it back up to the needed size in the following step involves zero-initialization. This means that ADIOS spends a lot of time per step just filling gigabytes of main memory with zeros. Quick measurements suggest a speedup down from ~1 minute for a data dump to ~30 seconds after fixing this.This PR quickly fixes the issue by adding conditionals to avoid shrinking the buffer.