CLBlast: Fix handling of on-device tensor data #3447
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fix uploading tensor data to device, including 3D, 4D, and non-contiguous tensors. Use correct offsets into data that is already in VRAM. Correct handling of OpenCL events when multiple commands are queued.
shibe2
changed the title
ggerganov
approved these changes
Oct 5, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #3307 and related issues.
Uploading was tested by reading data back from VRAM (for which there is no API). It now works properly in all cases.
Matrix multiplication works as well when
src0is already in VRAM. It allowed me to test broadcasting (#3402) in more configurations.Special code for matrix-vector multiplication remains broken. I intend to disable it in a separate request. But I can add it here too.
ggml_cl_mul(non-matrix multiplication) appears to have a problem with offsets too, but it is broken for other reasons as well, and I didn't touch it here.I used nullptr instead of NULL in some cases. Is it okay?