ggml : multi-thread ggml_rope() (~3-4 times faster on M1) #781
+37
−4
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Closed
prusnak
approved these changes
Apr 5, 2023
There was a problem hiding this comment.
Tested on M1.
Before the change (master 5a8c4f6) - second run:
llama_print_timings: load time = 2294.47 ms
llama_print_timings: sample time = 116.82 ms / 128 runs ( 0.91 ms per run)
llama_print_timings: prompt eval time = 2090.16 ms / 8 tokens ( 261.27 ms per token)
llama_print_timings: eval time = 23215.24 ms / 127 runs ( 182.80 ms per run)
llama_print_timings: total time = 25628.04 ms
After the change (commit 625f212) - second run:
llama_print_timings: load time = 1460.12 ms
llama_print_timings: sample time = 94.96 ms / 128 runs ( 0.74 ms per run)
llama_print_timings: prompt eval time = 1272.30 ms / 8 tokens ( 159.04 ms per token)
llama_print_timings: eval time = 21661.06 ms / 127 runs ( 170.56 ms per run)
llama_print_timings: total time = 23217.51 ms
howard0su
reviewed
Apr 5, 2023
| for (int64_t i3 = 0; i3 < ne3; i3++) { | ||
| for (int64_t i2 = (mode == 0 ? 0 : n_past); i2 < ne2; i2++) { | ||
| const int p = (mode == 0 ? n_past + i2 : i2); | ||
| for (int64_t i1 = 0; i1 < ne1; i1++) { | ||
| if (ir++ < ir0) continue; | ||
| if (ir > ir1) break; | ||
|
|
||
| for (int i0 = 0; i0 < n_dims; i0 += 2) { | ||
| const float theta = powf(10000.0, ((float)-i0)/n_dims); |
There was a problem hiding this comment.
theta can be calculated as theta *= factor in each loop. factor can be calculated out of the outer loop factor = powf(10000.0, ((float)-2)/n_dims); the initial theta = p.
There was a problem hiding this comment.
Open a PR if you observe performance gain improvement
|
How can i use Multi thread llama on cpu? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
🤖 Generated by Copilot at 625f212
Summary
🧶🚀🧑💻
This pull request improves the performance of the
ggmllibrary by parallelizing the rope operation on tensors, and modifying the graph executor to handle parallel tasks. It affects the fileggml.c.Walkthrough
f32andf16data types by dividing the input rows among the available threads (link, link, link, link)ggml_compute_forward_rope_f32andggml_compute_forward_rope_f16functions (link, link)ggml.cfile (link)