koboldcpp-1.23beta #179
Replies: 10 comments 24 replies
-
|
Please share your performance benchmarks with CLBlast GPU offloading. For good comparison, let us know:
|
Beta Was this translation helpful? Give feedback.
-
|
Here are my system specs:
Model: ggml-gpt4-x-alpasta-30b-q4_0.bin (MetalX) Prompt: Results: Subjectively, I can see that adding --gpulayers is putting stuff into my GPU's memory in 1.23. |
Beta Was this translation helpful? Give feedback.
-
|
Hmm so your generation speed improved but the processing speed got way worse after offloading, correct? Is this consistent with smaller models too? |
Beta Was this translation helpful? Give feedback.
-
|
Here's my attempt with a smaller model: ggml-gpt4-x-vicuna-13b-q5_1.bin Processing Prompt [BLAS] (221 / 221 tokens) 1.23 (no layers): Time Taken - Processing:6.9s (31ms/T), Generation:2.1s (261ms/T), Total:9.0s To me it looks like on 1.23, offloading layers to GPU doesn't impact processing time but has a minor performance increase during generation. |
Beta Was this translation helpful? Give feedback.
-
|
The way faster processing speed is likely just down to using cuBLAS for the processing rater than CLBlast , and nothing to do with the offloading code. Even before the offloading PR cuBLAS was known to be way faster on NVIDIA hardware than CLBlast for processing. To back up that assumption here are two runs on my setup with offloading off with version 1.22 vs 1.23
As you can see the generation speed is the same either way (within margin of error) but the processing is significantly shorter, and that is consistent between multiple runs. I've made a more detailed comment earlier with my full specs. |
Beta Was this translation helpful? Give feedback.
-
|
My system specs:
Model used: wizard-mega-13B.ggml.q5_1.bin, compiled koboldcpp from main branch as of today. Arguments: Results for generation of 100 tokens with 46 context tokens:
I think this GPU is just too slow and has too little VRAM for any meaningful speedup through OpenCL. 14 layers and above pretty much fills up all the available VRAM. |
Beta Was this translation helpful? Give feedback.
-
|
I have a very similar system (RX 580 4GB, but with an older Intel i5 4 core CPU) and came to the same conclusion. My times are very close to yours overall, too, just add about 60-90ms/T for my older CPU. |
Beta Was this translation helpful? Give feedback.
-
|
Trim sentences somehow broke with this update for me. Suddenly always trims partially completed sentences, irregardless of the setting. Smartcontext does not seem to mesh well with GPU offloading. Don't have time to re-run it, and might forget about it all together. I recommend people to benchmark at max context with smartcontext off vs on to verify their compatibility. |
Beta Was this translation helpful? Give feedback.
-
|
There shouldn't be any changes regarding sentence trimming. Perhaps you are in Adventure mode instead? |
Beta Was this translation helpful? Give feedback.
-
|
I'm not, it seems to be in any mode. I mostly use the default story mode, and sometimes chat with the chat GUI disabled. With trim off, it still trims sentences that are partially completed. I did however try older versions and it seems to have started in 1.21 and persisted since. |
Beta Was this translation helpful? Give feedback.
-
|
I just tried the latest version and it is working fine for me. I can generate incomplete sentences in story mode. Maybe your settings got corrupted. Can you try in a new incognito browser window? If that works, clear your cache, or disable "Persist Session" to reset everything to defaults. |
Beta Was this translation helpful? Give feedback.
-
I think I found the cause. I block lots of remote fonts and cosmetic mumbojumbo, which broke the functionality of the checkboxes, making them static. After I updated to a new version, the trim setting reset. While disabling it through the checkbox did untick the checkbox, it didn't actually disable it. Woops! Would be nice to have a config file without having to build form source, always easier to change settings that way. Actually, is it possible to have a separate config, then calling --config "./path/config.yaml" as a launch parameter? I do that with some other selfhosted crap on Linux that either defaults the config to strange locations that are hard to find, or requires different/multiple configs for different use cases. If not, could you possibly add that feature pretty please? |
Beta Was this translation helpful? Give feedback.
-
|
Config file wouldn't work as these settings are UI specific - they are set in the browser side. The API or koboldcpp doesn't even see them. |
Beta Was this translation helpful? Give feedback.
-
|
Specs
Test with
As you can see the generation time is slightly lower in 1.23, but the processing time is doubled, all of the tests were done with the exact same prompt. And the processing time difference is pretty consistent between runs and also applies to smaller models. As a person that usually have chats with very large contexts the processing time is a pretty big deal to me, especially if streaming mode is in use (it was not in use for these tests but I do sometimes use it) so it should be no surprise that my answer to the question Is Pepsi Okay? will have to be No, I'd prefer Coke please 😆. Also just out of curiosity is there a technical reason why |
Beta Was this translation helpful? Give feedback.
-
|
How does speeds compare to 1.21.3? |
Beta Was this translation helpful? Give feedback.
-
|
Both version are faster than 1.21.3.
Also one thing I forgot to mention, on build 1.23 the following is printed: The |
Beta Was this translation helpful? Give feedback.
-
|
The FP16 : 0 is an OpenCL limitation on Nvidia cards |
Beta Was this translation helpful? Give feedback.
-
|
Im assuming you are using the CUDA special edition for 1.22 (not self built) right? Blasbatchsize caps out because of memory limitations, too difficult to find a config that works well on all sizes. |
Beta Was this translation helpful? Give feedback.
-
Correct.
Okay, that's pretty understandable. It would just be really nice to be able to process the entire 2048 tokens in one batch. |
Beta Was this translation helpful? Give feedback.
-
|
Hi all, I've updated a new 1.23.1 version which has the fp16 set to disabled, as it appears that a majority of people find it slower than without. Please let me know if prompt processing with GPU layer offloading in 1.23.1 is still slower than 1.21.3. |
Beta Was this translation helpful? Give feedback.
-
|
I could use some clarification. Is the gpu offloading in this release different from the CUDA / cuBLAS version? |
Beta Was this translation helpful? Give feedback.
-
|
same concept, different backend |
Beta Was this translation helpful? Give feedback.
-
|
Ah, so this offloads layers using clblast and the "here is your cuda version now leave me alone" release uses cublas? Or does this release use a different library? Sorry if it's asking for stating the obvious but there is a lot of contrary info around. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, the normal builds will do GPU offloading via CLBlast (OpenCL), that is compatible with both AMD and Nvidia cards. The Cublas build was a special one time version that used CUDA instead. |
Beta Was this translation helpful? Give feedback.
-
|
Specs:
Cmd: koboldcpp-1.21.3\koboldcpp.exe --highpriority --smartcontext --useclblast 0 0 --usemirostat 2 5.0 0.1 spanielrassler_GPT4-X-Alpasta-30b-ggml/GPT4-X-Alpasta-30b_q4_0.bin
koboldcpp-1.22-CUDA-ONLY\koboldcpp_CUDA_only.exe --gpulayers 16 --highpriority --smartcontext --usemirostat 2 5.0 0.1 spanielrassler_GPT4-X-Alpasta-30b-ggml/GPT4-X-Alpasta-30b_q4_0.bin
koboldcpp-1.23beta\koboldcpp.exe --gpulayers 14 --highpriority --smartcontext --useclblast 0 0 --usemirostat 2 5.0 0.1 spanielrassler_GPT4-X-Alpasta-30b-ggml/GPT4-X-Alpasta-30b_q4_0.bin
koboldcpp-1.23.1\koboldcpp.exe --gpulayers 14 --highpriority --smartcontext --useclblast 0 0 --usemirostat 2 5.0 0.1 spanielrassler_GPT4-X-Alpasta-30b-ggml/GPT4-X-Alpasta-30b_q4_0.bin
koboldcpp-1.23.1\koboldcpp.exe --blasbatchsize 1024 --gpulayers 14 --highpriority --smartcontext --threads 6 --useclblast 0 0 --usemirostat 2 5.0 0.1 spanielrassler_GPT4-X-Alpasta-30b-ggml/GPT4-X-Alpasta-30b_q4_0.binTest: I ran 5 generations per model with the same prompts and calculated the averages. Layers were the highest number that wouldn't crash on load or during processing/generation. Results:
*: Ran it again with |
Beta Was this translation helpful? Give feedback.
-
|
Yes thanks for this. At least the generation time seems on par now. Unfortunately processing time still not as good, but already improved |
Beta Was this translation helpful? Give feedback.
-
Finally I can start to utilize my AMD GPU... └(^o^)┐ Specs
Models --gpulayers --useclblast 0 0 --blasbatchsize 1024 --threads 8 --smartcontext Results
7B without CLBlast
13B with CLBlast GPU offloading
13B without CLBlast
＼(＾O＾)／ |
Beta Was this translation helpful? Give feedback.
-
|
Very cool that you got a meaningful speedup! Though you do have a big honking GPU, haha. Have you tried lowering the CPU thread count to see if that gives you more tokens/second? Since it looks like that you are getting lower tokens/second than my R5 3600. Then again, part of that might be because of Windows. WSL2 can help, but I'm not sure if you can use your GPU + CLBlast with that. |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately opencl doesn't run on wsl/2 yet. |
Beta Was this translation helpful? Give feedback.
-
|
I think something is wrong in v1.23.1 In Instruct mode, click on Memory, change it to whatever you want, then submit a test instruction. Check the command prompt, you will see that your "memory" is there together with the default memory instruction right after, instead of overwriting the default. So I set the memory to "test" and send a simple instruct "hi". Here's my command prompt: I'm pretty sure the previous versions didn't do this, just fyi. I could be wrong I'd have to double check to be sure, but figure I'll let you know. Also, I'm not sure what version this started in, I only happened to notice it just now. |
Beta Was this translation helpful? Give feedback.
-
|
Looks like you accidentally enabled Instruct Mode but didn't mean to. You can change it under Settings > Format > Story Mode (or other), or uncheck the Instruction Prompt checkbox under that same menu. |
Beta Was this translation helpful? Give feedback.
-
|
Nope I don't think that's the case, I'm intentionally in Instruct mode. I just tested other versions - the problem started in 1.21.3 Kobold 1.20: Kobold 1.21.3: See the difference in the "prompt"? 1.21.3 just appends the default prompt "Below is an instruction that describes a task. Write a response that appropriately completes the request" even when I change "Memory" to "test". 1.20 doesn't do this - it uses whatever memory I set as the prompt, and doesn't add the default prompt right after. |
Beta Was this translation helpful? Give feedback.
-
|
Unchecking the Instruction Prompt checkbox doesn't remove it? |
Beta Was this translation helpful? Give feedback.
-
|
It does! Ok false alarm, I didn't notice that checkbox, it looks like it was introduced in 1.21.3. Prior to that, if you put anything in memory, it automatically removes the default. Now you explicitly have to uncheck the new checkbox to get rid of the default, regardless of whether you have something custom written in memory or not. That's fine, didn't realize that's how it works now. Thanks for pointing it out, my bad! |
Beta Was this translation helpful? Give feedback.
-
koboldcpp-1.23beta
A.K.A The "Is Pepsi Okay?" edition.
Changes:
--useclblast, combine with--gpulayersto pick number of layers to offloadTo use, download and run the koboldcpp.exe, which is a one-file pyinstaller.
Alternatively, drag and drop a compatible ggml model on top of the .exe, or run it and manually select the model in the popup dialog.
and then once loaded, you can connect like this (or use the full koboldai client):
http://localhost:5001
For more information, be sure to run the program with the
--helpflag.This release also includes a zip file containing the libraries and the
koboldcpp.pyscript, for those who prefer not use to the one-file pyinstaller.This discussion was created from the release koboldcpp-1.23beta.
Beta Was this translation helpful? Give feedback.
All reactions