[Question] Save internal state to disk

[ngxson]

Hi,

Thanks for your hard work on this project.

I've been playing with the code since few days. I'm trying to find a way to save the internal state of the model (or its context) that can be reused later. But until now it still doesn't work. I don't know if I'm missing something (I'm not good at all when talking about machine learning, I'm working more on system development.)

Here what I tried (inspired from code of main.cpp)

1. Load the model using llama_init_from_file
2. Call llama_eval on a prompt, for example " Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n"
3. Call llama_eval on a instruction, for example "### Instruction: Hi, I'm Xuan Son."
4. Save the kv_self buffer using llama_get_kv_cache => Expected: the saved data contains information about my name
5. Ask for what is my name?, the model correctly response that my name is Xuan Son
6. Exit the program
7. Re-run the program
8. Reload llama_init_from_file then kv_cache using llama_set_kv_cache => Expected: the loaded data contains information about my name
9. Ask for what is my name?, the model responses with nonsense words just love with 12 want some you, he saids

[chrfalch]

You need to save the n_past value and (if used) the n_tokens_past (used for repeat penalty calculations). You can see an example here: #730 (comment)

[ngxson]

I forgot to save last_n (or n_past). Thank you for pointing it out!

It works like a charm now.