[enhancement]: option to unload from memory

[aaronbolton]

Is there an existing issue for this?

• I have searched the existing issues

Contact Details

What should this feature add?

a command line option to unload model from RAM after a defined period of time

Alternatives

running as a container and using Sablier to shutdown the container after some time, this has the downside of if traffic isn't see through the web interface it will be shut even if jobs are running.

Additional Content

No response

[psychedelicious]

What's the use case for this?

[aaronbolton]

It would help with freeing up the memory for other applications such as Ollama, ollama has a similar feature were it unloads the models out of memory after 5min by default but this is configurable

[systemofapwne]

Invoke already (partially) unloads models from VRAM if you set memory bounds. However, this seems to only work properly, when you let the generation stop on its own - Not if you abort the queue.
I reported this already some time ago but devs seems to be busy with the huge amount of reported issues: #6759

[Jonseed]

@aaronbolton you might like my new Ollama Node for help expanding prompts in Invoke.

By the way, I have also noticed that Invoke doesn't seem to free up the memory after a generation, so Ollama runs slower on subsequent generations, probably because it is offloading most processing to cpu. It would be great if Invoke had an option to totally free the memory after each gen, for use cases like this. (My node has a toggle to unload the model from Ollama after generating the expanded prompt.)

[fahadshery]

I would like to second that.
Ollama unloads models from VRAM after a timeout (default 5 mins).
It would be nice to unload models from GPU once generation is completed and then wait for timeout and if there was no further generation then it unloads the model completely!
Currently I have to just restart invoke to unload the model from VRAM

[psychedelicious]

You can set lazy_offload: false in the invokeai.yaml config file and the app will actively offload models.

[fahadshery]

You can set lazy_offload: false in the invokeai.yaml config file and the app will actively offload models.

still gobbles up some VRAM when sitting idle. Is it not possible to fully unload?

[psychedelicious]

It only offloads down to the confused vram cache setting. There's no way at the moment to forcibly offload all models.

I think it'd be fairly straightforward to add an endpoint to do this, open to contributions.

[psychedelicious]

There is a button to clear the model cache in the queue tab

[lstein]

I am reopening this issue. I'm running ollama and invoke on the same server and I find their memory requirements are frequently clashing. It would be helpful to offer users the option to have the model cache automatically cleared after a fixed amount of inactivity. I would suggest the following:

1. Introduce a new config file option model_cache_keep_alive which specifies, in minutes, how long to keep a model in cache between generations. The default is 0, which means to keep the model in cache indefinitely, as is currently the case.
2. If no model generations occur within the timeout period, the model cache is cleared using the same backend code as the "Clear Model Cache" button in the queue tab.

I'm going to assign this to GitHub copilot, partly to test how well it can manage the Invoke code base.