Add exit call to interactive mode. #910
Conversation
|
I think it might be better to add It would require Signal handlers really ought to have a way to have a |
|
My first thought was to put this code into the signal handler, but I couldn't find where to place it. If you could point me in the right place that would be helpful!! |
|
Never mind I worked it out. You cannot pass a void to the signal handler. It needs a global ctx variable. I still think we need to have a way to pass commands to the input. It could do all kinds of things. Maybe we need to have some kind of prefix to the line to signal a command? For example For example, you could change the command line arguments in real time, then call stats print after a prompt. But maybe I'm over thinking it, as a reload of the program is pretty fast already now. |
|
This PR can probably be closed, as #924 includes the fix for signal handler stats print and also adds more functionality. Now the stats call is in the signal handler, we no longer have an "exit" command. |
* GradientAI Auto ROPE Base calculation https://gradient.ai/blog/scaling-rotational-embeddings-for-long-context-language-models has a formula that better fits the ideal rope scaling. Tested with Lllama3, checked calculation is correct for llama2. Retains logic for not scaling rope if under trained CTX. * add in solar scaling logic Solar based models require the context values to be multiplied by 8. This is (i'm guessing) because the positions as based on a 32k context, but sliding window of 4k. * Update model_adapter.h adding in tensor count to identify solar models based on tensor count of 435. * Update model_adapter.cpp add in n_tensor count for solar identification * refactor and cleanup GradientAI rope scaling --------- Co-authored-by: Concedo <39025047+LostRuins@users.noreply.github.com>
Problem:
The timing stats do not print when using "ctrl+c" to quit interactive mode. With all the problems I have been having with model speed I needed the stats to print out when I was finished running prompts.
Solution:
There is almost never a time the prompt is only a single word. So we check for input length equals 5, and if this is true check the word "exit" was entered in lower case.
This to me was the simplest solution to the problem, but maybe I am wrong. Maybe there is an easier way to handle this, I am open to discussion and changes. However using this solution its possible to add more commands to the program as required.