Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add OOM fail reason, attempted allocation size to exception messages #1827

Open
wants to merge 11 commits into
base: branch-25.04
Choose a base branch
from

Conversation

pmattione-nvidia
Copy link

@pmattione-nvidia pmattione-nvidia commented Feb 13, 2025

This PR addresses the following issues:

  1. Closes Issue 1791: It forwards on the details for why we got an OOM exception in try_to_expand(). It also does a more thorough job of forwarding on failure details in other locations. A test has been added to test this case explicitly.
  2. Closes Issue 1134: It adds the size of attempted allocations to the OOM and bad_alloc exceptions.

Notes:

  1. These code paths are already tested by the numerous EXPECT_THROW() macros already in the tests, and one more test was explicitly added.
  2. The modified loop in try_to_expand() in pool_memory_resource.hpp attempts to allocate for the case where try_size is less than min_size, whereas before it just immediately errored. If this isn't the behavior we want I can change it. Previously it was just erroring that we didn't have enough memory, which isn't right either.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@pmattione-nvidia pmattione-nvidia requested a review from a team as a code owner February 13, 2025 22:32
@github-actions github-actions bot added the cpp Pertains to C++ code label Feb 13, 2025
@pmattione-nvidia pmattione-nvidia marked this pull request as draft February 13, 2025 23:06
Copy link

copy-pr-bot bot commented Feb 13, 2025

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

Copy link
Member

@harrism harrism left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice job, getting better exception messages is valuable. A few suggestions.

@pmattione-nvidia
Copy link
Author

Note that this PR needs labels/etc. but I don't have permissions to add them.

@harrism harrism added non-breaking Non-breaking change bug Something isn't working labels Feb 21, 2025
Copy link
Member

@harrism harrism left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing!

Copy link
Contributor

@abellina abellina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vyasr vyasr requested a review from wence- February 22, 2025 01:45
@pmattione-nvidia pmattione-nvidia self-assigned this Feb 24, 2025
return block;
} catch (std::exception const& e) {
if (try_size <= min_size) {
RMM_LOG_ERROR("[A][Stream %s][Upstream %zuB][FAILURE maximum pool size exceeded: %s]",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(non-blocking) What does the [A] mean here? This may be a historical question for @harrism.

Copy link
Contributor

@bdice bdice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this work. All the changes look fine to me. Have you tested this downstream? Are you ready to merge?

@bdice
Copy link
Contributor

bdice commented Feb 24, 2025

@wence-, you requested changes -- could you provide feedback?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working cpp Pertains to C++ code non-breaking Non-breaking change
Projects
Status: Review
5 participants