[Version 0.2.50] Models get unloaded from memory after each query

[shebrien]

I am testing the llama 30B model and noticed that whenever I write a query it takes a long time to load the model to memory, then it writes the response, THEN it deallocates the used memory. I think this overhead will make it inefficient to use the web interface. This was also noticed when choosing alpaca 7B but the effect isn't as noticeable given the smaller model.

Would appreciate if there was an option to keep the models loaded to eliminate this overhad.

[yosr481]

+1 to this! I'm using version 0.3.1 and have experienced the same issue with models getting unloaded from memory after each query. It would be great if there was an option to keep the models loaded in memory to reduce the overhead and make it more efficient to use the web interface. Thanks for considering this feature request!

[trevtravtrev]

If this is true, and this currently loads the entire model into memory then deallocates after every prompt, I would recommend this being the #1 project priority to sort out. This is extremely inefficient and even makes the application unusable for any large weights or non-beefy PC's. Anyone have an idea how to tackle this?

[MasMedIm]

I have the same problem, any one can do a PR to solve that? This is a major issue.