btl/sm: rewrite of fast box (per-peer receive buffers) #13037
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lrbison
reviewed
Jan 21, 2025
bosilca
reviewed
Jan 21, 2025
There was a problem hiding this comment.
I don't see the need for moving code around like this, but I also don't see any issues with the reorder. I guess the only missing point is a performance comparison for a basic test (aka ping/pong) and then a fan-in (gather) and fan-out (bcast or scatter) in a heavily loaded setup (64 or 128 processes).
|
I ran this PR through some internal CI we have, and while it didn't collect performance data, it at least passed smoke test for correctness issues on Graviton3 (64-core neoverse-v1). |
hjelmn
force-pushed
the
clean_up_btl_sm_fbox_code_and_fix_edge_condition_that_can_cause_lost_messages
branch
from
4b352f2 to
3c512e6
Compare
bosilca
approved these changes
Feb 1, 2025
I was investigating possibly lost btl/sm messages and realized that the code is difficult to follow and it is not always clear what I was attempting to do. It is not clear if there is a problem but the rewrite is worth committing. This change does the following: - Seperate the fast box metadata out from the fast box receive data. These parts are logically separate so there was no need to keep adjusting the offset based on the metadata (start of buffer was offset 64, now 0). - Use modulo-math instead of toggling an extra bit to determine full vs empty. To keep this fast the modulo is done with bitwise-and with a mask and the fast box size has been limited to a power of two. This change simplifies the math and only has one special case to cover ( end overflow-- end less than start). - General cleanup of the code overall to improve readability. Signed-off-by: Nathan Hjelm <hjelmn@google.com>
hjelmn
force-pushed
the
clean_up_btl_sm_fbox_code_and_fix_edge_condition_that_can_cause_lost_messages
branch
from
3c512e6 to
95f7141
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I was investigating possibly lost btl/sm messages and realized that the code is difficult to follow and it is not always clear what I was attempting to do. It is not clear if there is a problem but the rewrite is worth committing.
This change does the following:
Seperate the fast box metadata out from the fast box receive data. These parts are logically separate so there was no need to keep adjusting the offset based on the metadata (start of buffer was offset 64, now 0).
Use modulo-math instead of toggling an extra bit to determine full vs empty. To keep this fast the modulo is done with bitwise-and with a mask and the fast box size has been limited to a power of two. This change simplifies the math and only has one special case to cover ( end overflow-- end less than start).
General cleanup of the code overall to improve readability.