-ctk q8_0 -ctv q8_0 return CUDA error

[aisensiy]

I run command like this:

./main -m qwen_72b_q3_k.gguf -n -1 -c 2048 -ctk q8_0 -ctv q8_0 -ngl 99 -i --multiline-input  -cml

It returns error like this:

llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 81/81 layers to GPU
llm_load_tensors:        CPU buffer size =   510.47 MiB
llm_load_tensors:      CUDA0 buffer size = 34467.06 MiB
..................................................................................................
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: freq_base  = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:      CUDA0 KV buffer size =  2720.00 MiB
llama_new_context_with_model: KV self size  = 2720.00 MiB, K (q8_0): 1360.00 MiB, V (q8_0): 1360.00 MiB
llama_new_context_with_model:  CUDA_Host input buffer size   =    21.02 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =   356.00 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =    16.00 MiB
llama_new_context_with_model: graph splits (measure): 2
CUDA error: invalid argument
  current device: 0, in function ggml_cuda_op_mul_mat at ggml-cuda.cu:10394
  ggml_cuda_cpy_tensor_2d(src0_dd_i, src0, i03, i02/i02_divisor, dev[id].row_low, dev[id].row_high, stream)
GGML_ASSERT: ggml-cuda.cu:255: !"CUDA error"
Aborted (core dumped)

But when I use f16 everything goes will.

[wooooyeahhhh]

The v cache cannot be quantised right now I believe. The k one can at q8_0 for some memory savings. So simply remove '-ctv q8_0' from the command.

[ngxson]

FYI, quantizing V can lead to very poor output result. Quantizing K does work with q4 and q8, but depends on backend.

[alice1bob]

Could you say why quantizing V cache leads to poor results? Did you do my measure? Why K cache won't do that then?

[ngxson]

I didn't do a test, but it was mentioned in the arg help message in an earlier version of llama.cpp, it was removed at some point (I don't know why)

Mathematically, because attention formula is softmax(...) * V, the V matrix has much significant impact on the output result