HDF5 MPI-collective I/O. #309
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1uc
force-pushed
the
1uc/hdf5-collective-reader
branch
from
248f364
to
d6bf8a9
Compare
We need the ability to read general hyperslabs into a 2D array.
The point is to split reading of the dataset into a separate function, and then make `resolve` safe for collective IO (assuming the newly introduced function is). The overload for reading a single `nodeID` is removed as it's unused now.
This commit refactors `_readSelection` in such a manner that: 1. Canonical selections don't require post-read shuffling. 2. The reading of the dataset is moved to a separate function. As a consequence, there's no need for an optimization for `std::string`, since those only mattern when reading from the `"@library"`, which should only happen indirectly. Hence, we free to ensure that those are always canonical.
This commit introduces the API for an Hdf5Reader. This reader abstracts the process of opening HDF5 files, and reading an `libsonata.Selection` from a dataset. The default reader calls the existing `_readSelection`.
This reader implements MPI-collective reading of datasets.
1uc
force-pushed
the
1uc/hdf5-collective-reader
branch
from
d6bf8a9
to
8695431
Compare
4 tasks
|
Has been moved to a separate repo. |
WeinaJi
pushed a commit
to BlueBrain/neurodamus
that referenced
this pull request
Jan 29, 2024
## Context When using `WholeCell` load-balancing, the access pattern when reading parameters during synapse creation is extremely poor and is the main reason why we see long (10+ minutes) periods of severe performance degradation of our parallel filesystem when running slightly larger simulations on BB5. Using Darshan and several PoCs we established that the time required to read these parameters can be reduced by more than 8x and IOps can be reduced by over 1000x when using collective MPI-IO. Moreover, the "waiters" where reduced substantially as well. See BBPBGLIB-1070. Following those finding we concluded that neurodamus would need to use collective MPI-IO in the future. We've implemented most of the required changes directly in libsonata allowing others to benefit from the same optimizations should the need arise. See, BlueBrain/libsonata#309 BlueBrain/libsonata#307 and preparatory work: BlueBrain/libsonata#315 BlueBrain/libsonata#314 BlueBrain/libsonata#298 By instrumenting two simulations (SSCX and reduced MMB) we concluded that neurodamus was almost collective. However, certain attributes where read in different order on different MPI ranks. Maybe due to salting hashes differently on different MPI ranks. ## Scope This PR enables neurodamus to use collective IO for the simulation described above. ## Testing <!-- Please add a new test under `tests`. Consider the following cases: 1. If the change is in an independent component (e.g, a new container type, a parser, etc) a bare unit test should be sufficient. See e.g. `tests/test_coords.py` 2. If you are fixing or adding components supporting a scientific use case, affecting node or synapse creation, etc..., which typically rely on Neuron, tests should set up a simulation using that feature, instantiate neurodamus, **assess the state**, run the simulation and check the results are as expected. See an example at `tests/test_simulation.py#L66` --> We successfully ran the reduced MMB simulation, but since SSCX hasn't been converted to SONATA, we can't run that simulation. ## Review * [x] PR description is complete * [x] Coding style (imports, function length, New functions, classes or files) are good * [ ] Unit/Scientific test added * [ ] Updated Readme, in-code, developer documentation --------- Co-authored-by: Luc Grosheintz <luc.grosheintz@gmail.ch>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR contains the code required to implement the MPI-collective reader. This will likely later be moved to it's own repository.