Use cupy to measure memory leak #777
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bdice
added
bug
grlee77
requested changes
Sep 6, 2024
dependencies.yaml
Outdated
| packages: | ||
| - cuda-python>=12.0,<13.0a0 | ||
| - matrix: {cuda: "11.*"} | ||
| packages: &test_cuda_python_cu11 |
There was a problem hiding this comment.
out of curiosity, what does this &test_cuda_python_cu11 notation do?
There was a problem hiding this comment.
This is a YAML anchor. We use this to define a list that we re-use in *test_cuda_python_cu11. We need a "fallback" list when the cuda selector is not defined.
Comment on lines
26
to
35
| def get_used_gpu_memory_mib(): | ||
| """Get the used GPU memory in MiB.""" | ||
| status, free, total = cuda.cudart.cudaMemGetInfo() | ||
| if status != cuda.cudart.cudaError_t.cudaSuccess: | ||
| raise RuntimeError("Failed to get GPU memory info.") | ||
| memory_used = (total - free) / (2**20) | ||
| return memory_used | ||
|
|
||
| status, num_gpus = cuda.cudart.cudaGetDeviceCount() | ||
| if status != cuda.cudart.cudaError_t.cudaSuccess or num_gpus == 0: |
There was a problem hiding this comment.
We can also potentially just use CuPy for this and avoid having to add the cuda-python dependency:
add up top:
import cupy as cpthen can use here:
def get_used_gpu_memory_mib():
"""Get the used GPU memory in MiB."""
dev = cp.cuda.Device()
free, total = dev.mem_info
memory_used = (total - free) / (2**20)
return memory_used
num_gpus = cp.cuda.runtime.getDeviceCount()
if num_gpus == 0:There was a problem hiding this comment.
Great. I'll implement that and remove the cuda-python dependency.
bdice
changed the title
jameslamb
approved these changes
Sep 6, 2024
|
Thanks @bdice for fixing the issue! |
gigony
approved these changes
Sep 6, 2024
|
/merge |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While adding support for Python 3.12 in #773, we found a problem where
GPUtildoes not support Python 3.12 (#775).This PR removes the dependency on
GPUtiland replaces it withcupy, which is already a dependency.Closes #775. Closes #776.