[FIX] Fix kernel bug #1959
Merged
[FIX] Fix kernel bug #1959
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I have completed all the kernel tests on the A800 GPU(x2), and all kernels can pass correctly.Can you review this PR? @WoosukKwon |
|
@WoosukKwon Are there any issues with this PR? |
WoosukKwon
reviewed
Dec 27, 2023
There was a problem hiding this comment.
Hi @jeejeelee thanks for submitting the PR. We've not noticed this bug since the device id is always 0 in vLLM. However, I agree that this change would make the kernel more portable.
Comment on lines
1
to
4
| @@ -1,21 +1,20 @@ | |||
| #include <torch/extension.h> | |||
| #include <ATen/cuda/CUDAContext.h> | |||
|
|
|||
| #include <torch/extension.h> | |||
| #include <c10/cuda/CUDAGuard.h> | |||
| #include "dispatch_utils.h" | |||
There was a problem hiding this comment.
style nit:
Suggested change
| #include <torch/extension.h> | |
| #include <ATen/cuda/CUDAContext.h> | |
| #include <torch/extension.h> | |
| #include <c10/cuda/CUDAGuard.h> | |
| #include "dispatch_utils.h" | |
| #include <torch/extension.h> | |
| #include <ATen/cuda/CUDAContext.h> | |
| #include <c10/cuda/CUDAGuard.h> | |
| #include "dispatch_utils.h" |
|
@WoosukKwon Thank you for your review. I have completed the following modifications:
Please review again |
WoosukKwon
approved these changes
Jan 3, 2024
There was a problem hiding this comment.
@jeejeelee LGTM! Thanks for the PR and apologies for the delayed review. We got many PRs last month and didn't have enough bandwidth due to the holidays. 😅
jedibrillo
pushed a commit
to jedibrillo/vllm
that referenced
this pull request
Jan 5, 2024
hongxiayang
pushed a commit
to hongxiayang/vllm
that referenced
this pull request
Feb 13, 2024
jinyouzhi
pushed a commit
to jinyouzhi/vllm
that referenced
this pull request
Sep 26, 2025
…m-project#1959) Signed-off-by: Youlei Yang <youlei.yang@intel.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While assessing the effectiveness of the RMSNorm operator, I observed that executing this operator on non-zero GPU resulted in a 'RuntimeError: CUDA error: an illegal memory access was encountered.'
Upon further investigation through debugging, I found that cause as the absence of device guards, most cuda kernels have the same issues .
I have addressed the issue by incorporating device guards for all kernels. Additionally, I have augmented the kernel tests by including device id,such as in the test_activationprovided