add PLaMo model

[okdshin]

This PR adds support of PLaMo-13B model.

PLaMo-13B is a LLaMA-based 13B model pre-trained on English and Japanese open datasets, developed by Preferred Networks, Inc. PLaMo-13B is released under Apache v2.0 license.

python convert-plamo-hf-to-gguf.py ./plamo-13b 1
./main -m ./plamo-13b/ggml-model-f16.gguf -p "これからの人工知能技術は" --temp 1.0 --top-p 0.95 --top-k 50 -n 32

[okdshin]

plamo-13b uses grouped query attention. I tried ggml_mul_mat(ctx0, K, Q); but it generates wrong values so I repeated K.
K : (128, 32, 5) -> K_repeated : (128, 32, 40)
Q : (128, 2, 40)

I thought llama2-70b could do it as it uses grouped query attention, but it didn't work as expected. Is there a misunderstanding in how to use ggml_mul_mat?

[ggerganov]

Broadcast in ggml_mul_mat is supported, so this repeat should not be necessary. LLaMA v2 70B definitely works.

I can't see an immediate problem in the implementation - will need to get the model and do some tests

[okdshin]

Thank you for your reply. I'll add the data that shows the difference in values and upload gguf files later.

[ggerganov]

As far as I can tell, this is exactly the same architecture as LLaMA v2. We should try to fit this into the existing arch implementation instead of creating a new arch. Whatever issues with the GQA might exist (i.e. convert script, broadcast, etc.), they should be identified and resolved as part of the LLM_ARCH_LLAMA

[okdshin]

Thank you for your feedback. To clarify, PLaMo-13B architecture differs from LLaMA v2 as it incorporates parallel layers, making it incompatible with the existing LLaMA implementation without significant modification.

(n_layer=1 for visualization)

[okdshin]

I uploaded plamo-13b gguf file. https://huggingface.co/sokada4/plamo-13b-gguf/tree/main

w/ K repeating

> ./bin/main -m plamo-13b-gguf/ggml-model-Q4_1.gguf -p "Hello my name is" --temp 0 -n 32
...
 Hello my name is sara and I'm a student at the University of Toronto. I have been playing piano for about five years now, but I started when I
...

w/o K repeating

> git diff                                                                               
diff --git a/llama.cpp b/llama.cpp
index 9868c81..0ba534c 100644
--- a/llama.cpp
+++ b/llama.cpp
@@ -6007,12 +6007,14 @@ static struct ggml_cgraph * llm_build_plamo(
             ggml_set_name(K, "K");
 
             // K * Q
-            //struct ggml_tensor * KQ = ggml_mul_mat(ctx0, K, Q);
+            struct ggml_tensor * KQ = ggml_mul_mat(ctx0, K, Q);
             // we should avoid to repeat K but current ggml_mul_mat generates wrong values for grouped query att 
+            /*
             struct ggml_tensor * K_repeated = ggml_new_tensor_3d(ctx0, GGML_TYPE_F32, K->ne[0], K->ne[1], Q->ne[2]);
             offload_func_kq(K_repeated);
             ggml_set_name(K_repeated, "K_repeated");
             struct ggml_tensor * KQ = ggml_mul_mat(ctx0, ggml_repeat(ctx0, K, K_repeated), Q);
+            */
             offload_func_kq(KQ);
             ggml_set_name(KQ, "KQ");

> ./bin/main -m plamo-13b-gguf/ggml-model-Q4_1.gguf -p "Hello my name is" --temp 0 -n 32
...
 Hello my name is not a .вЖ, and then





</p>
...

[okdshin]

@ggerganov Excuse me. Please tell me what I have to do to merge this PR.

[cebtenzzre]

Excuse me. Please tell me what I have to do to merge this PR.

I can't promise anything, but it would help if you resolved the merge conflicts.

[ggerganov]

@okdshin The model graph code in llama.cpp has been recently refactored (#3837) in order to reduce code duplication and make things more consistent. To merge this PR, it will have to be reworked to follow the new graph building pattern that is used for all other models.

[okdshin]

@cebtenzzre @ggerganov Thank you for your comments. I will modify this PR to follow the new graph building pattern.

[okdshin]

@ggerganov @cebtenzzre I reworked to follow the new graph building pattern. Could you check it?

[ggerganov]

I was curious why the broadcast does not work here, so I tested the model and the following patch makes it work on the CPU, using the original llm_build_kqv:

diff --git a/ggml.c b/ggml.c
index ad546a73..026c2205 100644
--- a/ggml.c
+++ b/ggml.c
@@ -9650,7 +9650,7 @@ static void ggml_compute_forward_mul_mat(
             for (int64_t i12 = 0; i12 < ne12; i12++) {
                 // broadcast src0 into src1 across 2nd,3rd dimension
                 const int64_t i03 = i13/r3;
-                const int64_t i02 = i12/r2;
+                const int64_t i02 = i12%5;
 
                 const void  * x = (char *)            src0->data + i02*nb02 + i03*nb03;
                 const float * y = (float *) ((char *) src1->data + i12*nb12 + i13*nb13);
@@ -9761,7 +9761,7 @@ static void ggml_compute_forward_mul_mat(
 
                 // broadcast src0 into src1
                 const int64_t i03 = i13/r3;
-                const int64_t i02 = i12/r2;
+                const int64_t i02 = i12%5;
 
                 const int64_t i1 = i11;
                 const int64_t i2 = i12;

So it means we need during the conversion to shuffle the attn_q.weight tensors to match the expected order in llama.cpp. This will avoid having to implement the custom plamo_llm_build_kqv function.

[ggerganov]

I pushed the WIP version here: https://github.com/ggerganov/llama.cpp/tree/gg/plamo-test

This one works without explicit ggml_repeat, but obviously it's not OK because it includes the patch in ggml.c that I posted above. The goal is to update the convert script so that it shuffles the necessary tensors in such a way that we don't have to modify the broadcast logic in ggml.c. Hope this helps

[okdshin]

Thank you very much! I will modify the conversion script and try to avoid to use the customized llm_build_kqv.

[okdshin]

I modified to shuffle attn_q.weight and attn_output.weight for the broadcasting and removed plamo_llm_build_kqv. Could you review again?

[ggerganov]

Good job! Thank you for following up with all the necessary changes!