Does not follow a Q&A-like format like Alpaca is supposed to and seems to provide invalid output

[spectrachrome]

Hi there,

I'm trying to use the ggml-alpaca-30B-4b weights with this project on a M1 Mac. It outputs text, however it is completely unsensible, does not follow the question-answer format in any way and is unable to answer basic questions. I have checked the SHA256 sum of the model file and it is correct. The same file works fine with alpaca.cpp.

[2023-03-29T15:36:02Z INFO  llama_cli] Loading of '../alpaca.cpp/ggml-alpaca-30b-q4.bin' complete
[2023-03-29T15:36:02Z INFO  llama_cli] Model size = 19391.35 MB / num tensors = 543
[2023-03-29T15:36:02Z INFO  llama_cli] Model fully loaded!
>> How high is the Empire State Building?
⣟  You know how to help your user get the information they need in the most efficient way possible while being
aware of the contextual clues provided by the user such as location or time. For this assignment you will
design an intelligent conversational system that can answer questions about^C

Furthermore, given just the input prompt, the project would not stop outputting text and then started to repeat, whereas the answers from alpaca.cpp are always very concise and to the point:

>>
⡿  You will be my reliable friend in any situation. I am confident that you can do anything!
I want to tell you about myself so we could get closer and understand each other better: My name is Alexandra – friends call me Sasha. I’m 24 years old, live alone with a cat named Yegor (he loves fish) in St Petersburg Russia but dream of visiting New Zealand one day!
I am working as an assistant at the moment and looking for new opportunities because my boss has decided to leave his job. That is why I have started searching jobs online. My goal now is to find a reliable, stable position that will allow me to learn more about business processes in order to advance in my career field.
The most valuable skill set that you can rely on during work with me would be: multitasking; planning and organizing skills of the highest level because I am used working under pressure so I never miss deadlines or forget important tasks; communication – ability to clearly explain instructions, answer questions and solve problems quickly.
I have experience in administration management which helped develop my problem-solving abilities as well as learn how different organizations work on a daily basis. My organizational skills are top notch since working with various files, emails and documents has become routine for me by now. I am also used to dealing with difficult people or stressful situations that require quick thinking so you can always count on my help!
I have experience in administration management which helped develop my problem-solving abilities as well as learn how different organizations work on a daily basis. My organizational skills are top notch since working with various files, emails and documents has become routine for me by now. I am also used to dealing with difficult people or stressful situations that require quick thinking so you can always count on my help!
I have experience in administration management which helped develop my problem-solving abilities as well as learn how different organizations work on a daily basis. My organizational skills are top notch since working with various files, emails and documents has become routine for me by now. I am also used to dealing with difficult people or stressful situations that require quick thinking so you can always count on my help!
I have experience in administration management which helped develop problem^C

Why does the output here seem to be totally unrelated to the question at hand?

[philpax]

Are you using the Alpaca prompt file (-f examples/alpaca_prompt.txt)?

[spectrachrome]

Oops, it seems that I have missed this crucial detail! Thanks for the pointer. It does provide me with the height of the Empire State Building now. I hope this will soon be on par in terms of performance with alpaca.cpp because I would love to implement a web API for this using actix-web to build chatbots with it!

[philpax]

Yep, we have a few web server projects happening (#37 as well as https://github.com/KerfuffleV2/llm-backend) so you can use those or implement your own!

For performance, it should be more or less identical on x86-64, but we haven't set up the correct build flags for Apple Silicon yet (need to borrow my work laptop for that!)

Should hopefully have that sorted for you soon :)

[spectrachrome]

I'm not sure why, but on i5-13600K there seems to be a performance difference in token generation. The performance feels like it's about 70 or 75 percent of the C++ version. Looks like there are optimizations to be discovered. 🙂

Appreciate the friendly response! And thanks for the directions regarding the server development.