Try whether OpenLLaMa works

[prusnak]

... or whether we need to tweak some settings

GitHub: https://github.com/openlm-research/open_llama

HuggingFace: https://huggingface.co/openlm-research/open_llama_7b_preview_300bt

---

edit: GGML models uploaded to HH by @vihangd => https://huggingface.co/vihangd/open_llama_7b_300bt_ggml

[Green-Sky]

Other than the 7B model, we are also training a smaller 3B model in hope of facilitating language model usage in low resource use cases.

sounds good.

I see inference tests in the CI coming

[Green-Sky]

./main -m models/open_llama_7b_preview_200bt/ggml-model-q5_1.bin -n 100 -s 3
main: build = 491 (7dffb0d)
main: seed  = 3
llama.cpp: loading model from models/open_llama_7b_preview_200bt/ggml-model-q5_1.bin
llama_model_load_internal: format     = ggjt v1 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 512
llama_model_load_internal: n_embd     = 4096
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 32
llama_model_load_internal: n_layer    = 32
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 9 (mostly Q5_1)
llama_model_load_internal: n_ff       = 11008
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size =  68.20 KB
llama_model_load_internal: mem required  = 6612.58 MB (+ 1026.00 MB per state)
llama_init_from_file: kv self size  =  256.00 MB

system_info: n_threads = 12 / 24 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |
sampling: repeat_last_n = 64, repeat_penalty = 1.100000, presence_penalty = 0.000000, frequency_penalty = 0.000000, top_k = 40, tfs_z = 1.000000, top_p = 0.950000, typical_p = 1.000000, temp = 0.800000, mirostat = 0, mirostat_lr = 0.100000, mirostat_ent = 5.000000
generate: n_ctx = 512, n_batch = 512, n_predict = 100, n_keep = 0


 2016, the company has been awarded a contract to build the city's fifth water plant. The project is being constructed by NEPCO Constructors, which was recently acquired by Bechtel. The company was selected in May of 2016 to build a new wastewater treatment facility for the city and is also currently working on construction of the city's first solar farm. The solar array will supply energy to water utility operations at the site. "This project exemplifies
llama_print_timings:        load time =   421.38 ms
llama_print_timings:      sample time =    69.40 ms /   100 runs   (    0.69 ms per run)
llama_print_timings: prompt eval time =   206.36 ms /     2 tokens (  103.18 ms per token)
llama_print_timings:        eval time = 13842.44 ms /    99 runs   (  139.82 ms per run)
llama_print_timings:       total time = 14355.46 ms

[Green-Sky]

(for 200bt)

$ sha256sum *.bin
9119b65346b0b503b29e04d75ca444c626d7c3c5f886ef7a52aa67ac44102d60  ggml-model-f16.bin
ec0d36db9435481b91c6c8e351406a16b19ce328334a64fc27bd52b1c9a3f3d6  ggml-model-q5_1.bin

The fact that this new model works here is great as it means that we can move beyond the leaked Llama into a truly open model while keeping the existing GGML code and applications.

@Green-Sky were any changes to llama.cpp/convert.py required to get this model to load?

[Green-Sky]

Ok, so, Perplexity seems to spiral out of control. Something must be wrong.
[1]11.4237,[2]275.3643,[3]942.8642,[4]1526.5208,[5]1918.0205,[6]2343.1441

@eiery no, no changes, but I made sure to call the convert.py from the model subdirectory, to make sure it picks the correct tokenizer.

[ggerganov]

@Green-Sky Should be fixed by bf4b22f

[Green-Sky]

@ggerganov I was on bf4b22f

$ bin/perplexity -m ../models/open_llama_7b_preview_200bt/ggml-model-q5_1.bin -f ../wikitext-2-raw/wiki.test.raw
main: build = 489 (bf4b22f)
main: seed  = 1683073191
llama.cpp: loading model from ../models/open_llama_7b_preview_200bt/ggml-model-q5_1.bin
llama_model_load_internal: format     = ggjt v1 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 512
llama_model_load_internal: n_embd     = 4096
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 32
llama_model_load_internal: n_layer    = 32
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 9 (mostly Q5_1)
llama_model_load_internal: n_ff       = 11008
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size =  68.20 KB
llama_model_load_internal: mem required  = 6612.58 MB (+ 1026.00 MB per state)
llama_init_from_file: kv self size  =  256.00 MB

system_info: n_threads = 12 / 24 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 |
perplexity : calculating perplexity over 616 chunks, batch_size=512
31.14 seconds per pass - ETA 5 hours 19 minutes
[1]11.4237,[2]275.3643,[3]942.8642,[4]1526.5208,[5]1918.0205,[6]2343.1441,[7]2831.9216,[8]3417.0539,[9]3617.7688,[10]3698.6451,[11]4032.9414,

it was segfaulting before 😄

[SlyEcho]

It seems to break down after resetting the context.

[ggerganov]

The OpenLLaMA generation fails when the prompt does not start with the BOS token 1.
For main a workaround is to use --keep 1 or more. This will guarantee that during context swap, the first token will remain BOS.

For perplexity - there is no workaround. The fix is to change the chunks to always start with BOS token.
I am currently evaluating how this affects the existing perplexity results for LLaMA.

Does anyone know if OpenLLaMA's behavior is correct?
I mean, shouldn't it still work, even if the first token is not BOS?

---

LLaMA (vanilla)

Model	Format	PPL (Original)	PPL (With BOS)
7B	F16	5.9565	5.9066
7B	Q4_0	6.2103	6.1621
7B	Q5_1	5.9934	5.9481

[Green-Sky]

Note that we use BOS (beginning of sentence) token (id=1) during training, so it is important to prepend this token for best performance during few-shot evaluation.

but why waste an extra slot in context?

[ggerganov]

but why waste an extra slot in context?

I guess it makes the generation more accurate. I think it depends on how was the training performed.
If all training sequences had a BOS at the start, then we have to satisfy this requirement during inference to get a correct result.

The good news is that the fix is trivial
The bad news is that we need to redo all Perplexity calculations