Feature: general purpose data distribution testing to support non-uniform data #129
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Copy h5bench_write_normal_dist.c as baseline.
Created a new benchmark test name and added selection support into h5bench.py test wrapper.
Added parsing of DATA_DIST_PATH into params struct to record the data distrbution based on an input file.
Takes the provided DATA_DIST_PATH file and reads the data sizes per rank from the file feeding into holder array.
Create make targets and add file to install list.
Remove rank 0 output summary limit so all processes report performance rather than just accepting rank 0 results. This is especially important for non-uniform data distribution but should be considered for uniform since no summary stats are computed. Recommend --tag-output to track per-rank stdout. Remove naive total size compute and limit it to a per rank size value.
Add scaling paramter to tests to scale particle count in order to create memory footprints that more accurately reflect a data distribution. Data distribution inputs are in particle counts. Particles are 32-byte structures so a data distribution measured in bytes needs to be scaled down so the particals instiated match the actual data footprint, in multiples of 32-byte particles.
Change the deployed binary to "write_var_data_dist" so it matches the configured test reference in h5bench. The binary name needs to match the test name so that it can be called by the hbench wrapper. This follows convention of "write_var_normal_dist". Update code to log the correct benchmark name.
Change action/artifacts-upload from v2 to v4 to remove dependence on deprecated v2. The v4 syntax for the artifact-upload remains the same so a simple update of the version number should be sufficient. See blog post for details: https://github.blog/changelog/2024-02-13-deprecation-notice-v1-and-v2-of-the-artifact-actions/
Update to latest version of container to see if it avoids the missing distutils dependency reported with the @0.11 version.
…ama/h5bench into feat-data-dist Grab the github actions fixes to ensure current feature branch validation tests run successfully.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adding a feature to support data distributions that express non-uniform workloads for HDF5 performance testing. This builds on the h5bench_write_normal_dist.c approach and extends it to allow per-rank data footprint specification via a data distribution configuration file.
Additional extensions to h5bench to allow selection and configuration of the new h5bench_data_dist.c test that encapsulates this feature.