[Cuda] Skip FreeDataSpace when CUDA driver is in inconsistent state #16980
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Prior to this commit, the RAII handler in `NDArray` would always attempt to free a cuda memory allocation on destruction. However, the call to `cudaFree` may throw an exception. If this happens during stack unwinding due to a previously-thrown exception, this causes the program to immediately terminate, making it difficult to identify the source of the original error. This can commonly occur if an async compute kernel performs an illegal memory access. An exception is thrown from the next cuda API call following the asynchronous error, causing the stack to unwind. If the stack contains any `NDArray` instances which reference cuda allocations, the destructor of these `NDArray` instances will attempt to free memory, triggering the segfault. This commit updates the `CUDADeviceAPI::FreeDataSpace` function to check if the program is currently unwinding the stack due to a thrown exception, while the cuda driver has been left in an unrecoverable state. If this occurs, no attempt to free memory is made, as all cuda API calls will result in an error, and the original exception is allowed to propagate. If the cuda driver is in an unrecoverable state, but no exception is currently unwinding the stack, then this may be the first cuda API call to occur after the asynchronous error. In this case, the `cudaFree` call is still performed, which throws the initial exception.
masahi
approved these changes
May 13, 2024
masahi
reviewed
May 13, 2024
| @@ -142,6 +142,24 @@ class CUDADeviceAPI final : public DeviceAPI { | |||
| } | |||
|
|
|||
| void FreeDataSpace(Device dev, void* ptr) final { | |||
| if (std::uncaught_exceptions() && cudaPeekAtLastError() == cudaErrorIllegalAddress) { | |||
There was a problem hiding this comment.
I assume that uncaught_exceptions() is not going to add any overhead.
There was a problem hiding this comment.
Should be minimal compared to the memory allocation/free itself. The exact implementation of std::uncaught_exceptions() is compiler-dependent, but it typically is implemented as a read of a static thread-local variable.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Prior to this commit, the RAII handler in
NDArraywould always attempt to free a cuda memory allocation on destruction. However, the call tocudaFreemay throw an exception. If this happens during stack unwinding due to a previously-thrown exception, this causes the program to immediately terminate, making it difficult to identify the source of the original error.This can commonly occur if an async compute kernel performs an illegal memory access. An exception is thrown from the next cuda API call following the asynchronous error, causing the stack to unwind. If the stack contains any
NDArrayinstances which reference cuda allocations, the destructor of theseNDArrayinstances will attempt to free memory, triggering the segfault.This commit updates the
CUDADeviceAPI::FreeDataSpacefunction to check if the program is currently unwinding the stack due to a thrown exception, while the cuda driver has been left in an unrecoverable state. If this occurs, no attempt to free memory is made, as all cuda API calls will result in an error, and the original exception is allowed to propagate.If the cuda driver is in an unrecoverable state, but no exception is currently unwinding the stack, then this may be the first cuda API call to occur after the asynchronous error. In this case, the
cudaFreecall is still performed, which throws the initial exception.