-
Notifications
You must be signed in to change notification settings - Fork 208
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add OOM fail reason, attempted allocation size to exception messages #1827
base: branch-25.04
Are you sure you want to change the base?
Add OOM fail reason, attempted allocation size to exception messages #1827
Conversation
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice job, getting better exception messages is valuable. A few suggestions.
Note that this PR needs labels/etc. but I don't have permissions to add them. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for fixing!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @pmattione-nvidia
return block; | ||
} catch (std::exception const& e) { | ||
if (try_size <= min_size) { | ||
RMM_LOG_ERROR("[A][Stream %s][Upstream %zuB][FAILURE maximum pool size exceeded: %s]", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(non-blocking) What does the [A]
mean here? This may be a historical question for @harrism.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for this work. All the changes look fine to me. Have you tested this downstream? Are you ready to merge?
@wence-, you requested changes -- could you provide feedback? |
This PR addresses the following issues:
try_to_expand()
. It also does a more thorough job of forwarding on failure details in other locations. A test has been added to test this case explicitly.Notes:
EXPECT_THROW()
macros already in the tests, and one more test was explicitly added.try_to_expand()
inpool_memory_resource.hpp
attempts to allocate for the case wheretry_size
is less thanmin_size
, whereas before it just immediately errored. If this isn't the behavior we want I can change it. Previously it was just erroring that we didn't have enough memory, which isn't right either.Checklist