How to make it faster?

[theytookcake]

Hello!

What settings should I tweak to make it faster? I dont know if its even possible.

I know that smaller models answer faster, But for that I need I'm using a mistral 7b instruct model and it takes around 10 seconds to answer. Anything I can tweak to make it answer faster?

[amakropoulos]

Hi! 10 seconds is too long.
Is this the time it takes to get the first response or every response?
In a 8-year old 6-core CPU I have it takes ~2-3 seconds for the response to arrive.
Can you build the project and see if it takes the same amount of time?

[theytookcake]

Hello! I'm now running the WarmUp Function in the Start and its much faster, using stream it takes around a second for text to start arriving!

[amakropoulos]

Perfect! Yes, that was the reason I was asking :).

The first response needs to process the character prompt, and if it is large it takes some time.
That's why the Warmup helps. It processes the character prompt and the result is cached.
I should added it in all the samples!

I'll close the issue, let me know if you get stuck with anything else!