Hierarchical Model LRU/ARC and Manual Reclaim

[sempervictus]

Is your feature request related to a problem? Please describe.
Models remain in GPU memory forever, especially in distributed deployments where shutdown API does not propagate.

Describe the solution you'd like

1. Permit (appropriately privileged) user eviction of models from GPU memory into system RAM or fully (exit their runner) to disk across the fleet, on a node-by-node or fleet-wide basis through standard API interfaces
2. Implement "ageing" for model states in memory tracking the last interaction with their consumer to proactively free resources for new tasks incoming by demoting layers to system memory or disk. LRU is the simplest way to do this but frequency of model use does have relevance, so an Adaptive Replacement Cache (ARC) pattern might be more appropriate.
3. Implement "tuning knobs" to bias behavior such as demoting small models first or entirely evicting certain ones based on their metadata instead of trying to evict them to RAM.

Describe alternatives you've considered

• Currently looking at modifying an openwebui contribution to provide "user-friendly eviction" capability via the shutdown API

Additional context

• Full implementation may merit addition of metadata fields to local models or maintenance of state for the known set to reflect their eviction behaviors

[mudler]

good points here, and lots of room for improvements. currently we have a fairly basic handling by setting LOCALAI_SINGLE_ACTIVE_BACKEND to true - this will basically make sure to have only a single model loaded every time, but this calls for better handling.

Permit (appropriately privileged) user eviction of models from GPU memory into system RAM or fully (exit their runner) to disk across the fleet, on a node-by-node or fleet-wide basis through standard API interfaces

This is more or less available already. we have a shutdown API defined here:

LocalAI/core/http/routes/localai.go

Line 74 in 11c67d1

router.Post("/v1/backend/shutdown", localai.BackendShutdownEndpoint(backendMonitorService))

. However, this does currently not take account if we are e.g. federated or not. In that case we might want to implement first a mechanism to syncronize node states (which should be also used to setup the models in the first place across the fleet).

Implement "ageing" for model states in memory tracking the last interaction with their consumer to proactively free resources for new tasks incoming by demoting layers to system memory or disk. LRU is the simplest way to do this but frequency of model use does have relevance, so an Adaptive Replacement Cache (ARC) pattern might be more appropriate.

This is something I really like to see. Currently the LOCALAI_SINGLE_ACTIVE_BACKEND heuristic is simple, but now we should be able to pull data from the host such as GPU used, which can be used to decide which model to unload.

Implement "tuning knobs" to bias behavior such as demoting small models first or entirely evicting certain ones based on their metadata instead of trying to evict them to RAM.

Maybe metadatas applied on the model config would be a way to tackle this. LocalAI then would act as a scheduler by reading annotations on the models which dictate behavior. In this way we can also extend it down the line with different annotations.

[sempervictus]

Thank you for the detailed response @mudler, apologies for the lag on my end.
The shutdown endpoint is ... "biased" 😉 - it appends .bin to the end of model names requested for eviction :-(

I modified the openwebui "unload models from vram" function to try and actuate this endpoint but i get errors about the suffix being added and the endpoint being wrong as a result. Otherwise, that's a somewhat workable solution for the moment.

12:30AM ERR Server error error="model all-MiniLM-L6-v2.bin not found" ip=172.18.0.20 latency="96.243µs" method=POST status=500 url=/v1/backend/shutdown
12:30AM ERR Server error error="model google_gemma-3-12b-it-qat-Q4_0.gguf.bin not found" ip=172.18.0.20 latency="59.48µs" method=POST status=500 url=/v1/backend/shutdown
...
12:31AM ERR Server error error="model mmproj-google_gemma-3-12b-it-qat-f16.gguf.bin not found" ip=172.18.0.20 latency="52.992µs" method=POST status=500 url=/v1/backend/shutdown

Re state-sync: something like RAFT/NATS is probably a good idea with a thin local persistence tier via sqlite or rocks to recover federated topologies after outages. Big fan of your VPN trick, i'm one of the metasploit people and this sort of whitehat pivoting technique warms my heart. That said, we need to be able to generate federation token with consideration for NAT and other network oddities (feed the correct seed values in for peer discovery and authorization)

I like the model-level metadata concept, we can vectorize that and pass it through analysis layers since this thing runs analysis layers 😄

[BjoKaSH]

Implement "ageing" for model states in memory tracking the last interaction with their consumer to proactively free resources for new tasks incoming by demoting layers to system memory or disk. LRU is the simplest way to do this but frequency of model use does have relevance, so an Adaptive Replacement Cache (ARC) pattern might be more appropriate.

This is something I really like to see. Currently the LOCALAI_SINGLE_ACTIVE_BACKEND heuristic is simple, but now we should be able to pull data from the host such as GPU used, which can be used to decide which model to unload.

Implement "tuning knobs" to bias behavior such as demoting small models first or entirely evicting certain ones based on their metadata instead of trying to evict them to RAM.

Maybe metadatas applied on the model config would be a way to tackle this. LocalAI then would act as a scheduler by reading annotations on the models which dictate behavior. In this way we can also extend it down the line with different annotations.

I think for the beginning even a very simple "auto-unload after x minutes" would already be a significant improvement for all those of us who run LocalAI for private / home use on a single computer with maybe an NVIDEA 4070 or similar just-so-affordable GPU for acceleration.

For example, I am running the LocalAI All-in-One GPU docker image on a box with 64 GB system RAM, 24 CPU cores (in one physical Intel i7-13700T) and a 4070 with 12 GB of vRAM. The LocaklAI instance is mostly used as back-end for Nextcloud's AI function for the self-hosted family cloud and an occasional chat for text processing.

In that configuration LocalAI is barely usable. With luck, it fits two models in the GPU, completely occupying it. So basically every time I want to switch model (or use some other software that needs GPU resources, like 3D modelling and rendering) I have to stop the LocalAI docker container and restart it afterwards to have space for the other model or other program.

An (in my case) "aggressive" auto-unload after e.g. 10 minutes would help a lot.

Going for some more intelligent memory management as proposed by sempervictus, which would allow LocalAI to actively evict a model when loading another fails due to memory shortage, would of course be better, but even a simple, unconditional "unload after x minutes idle time" would really make life easier.

[mudler]

Implement "ageing" for model states in memory tracking the last interaction with their consumer to proactively free resources for new tasks incoming by demoting layers to system memory or disk. LRU is the simplest way to do this but frequency of model use does have relevance, so an Adaptive Replacement Cache (ARC) pattern might be more appropriate.

This is something I really like to see. Currently the LOCALAI_SINGLE_ACTIVE_BACKEND heuristic is simple, but now we should be able to pull data from the host such as GPU used, which can be used to decide which model to unload.

Implement "tuning knobs" to bias behavior such as demoting small models first or entirely evicting certain ones based on their metadata instead of trying to evict them to RAM.

Maybe metadatas applied on the model config would be a way to tackle this. LocalAI then would act as a scheduler by reading annotations on the models which dictate behavior. In this way we can also extend it down the line with different annotations.

I think for the beginning even a very simple "auto-unload after x minutes" would already be a significant improvement for all those of us who run LocalAI for private / home use on a single computer with maybe an NVIDEA 4070 or similar just-so-affordable GPU for acceleration.

For example, I am running the LocalAI All-in-One GPU docker image on a box with 64 GB system RAM, 24 CPU cores (in one physical Intel i7-13700T) and a 4070 with 12 GB of vRAM. The LocaklAI instance is mostly used as back-end for Nextcloud's AI function for the self-hosted family cloud and an occasional chat for text processing.

In that configuration LocalAI is barely usable. With luck, it fits two models in the GPU, completely occupying it. So basically every time I want to switch model (or use some other software that needs GPU resources, like 3D modelling and rendering) I have to stop the LocalAI docker container and restart it afterwards to have space for the other model or other program.

An (in my case) "aggressive" auto-unload after e.g. 10 minutes would help a lot.

Note that is already possible by setting LOCALAI_WATCHDOG_IDLE=true which would make LocalAI kill automatically idle-ing backends after a specific amount of time (defaults to 15m)

Going for some more intelligent memory management as proposed by sempervictus, which would allow LocalAI to actively evict a model when loading another fails due to memory shortage, would of course be better, but even a simple, unconditional "unload after x minutes idle time" would really make life easier.

One of the most complex side of this is that in order to do proper eviction we should have an estimation of how much VRAM a model would take. This is hard to achieve on different backends and have a "generic" enough solution.