Add option for LRU eviction on memory pressure / load #9375

Activity

1. Xnyle

   added

   enhancementNew feature or requestNew feature or request

   on Apr 16, 2026
2. Xnyle

   changed the title [-]Add option for LRU eviction on memory pressure[/-] [+]Add option for LRU eviction on memory pressure / load[/+] on Apr 16, 2026
3. SuperMarioYL

   added a commit that references this issue on Apr 26, 2026

   feat(watchdog): add size-aware LRU eviction mode

   ...

   This issue will close once commit bf9228a is merged into the 'master' branch.

   bf9228a
4. walcz-de commented on Apr 28, 2026

   

   walcz-de

   on Apr 28, 2026

   Contributor

   More actions

   Note for anyone hitting this: the feature is actually already implemented as the memory_reclaimer (added in #7583, Dec-16-2025). You can enable it via LOCALAI_MEMORY_RECLAIMER=true and LOCALAI_MEMORY_RECLAIMER_THRESHOLD=0.95 (or via the runtime-settings JSON / Settings UI). When GPU usage exceeds the threshold, the watchdog evicts LRU models.

   However, there's a sharp corner that makes it unusable on llama-cpp setups today. Surfaced cleanly on a single-GPU AMD Strix Halo (gfx1151, 96 GB unified VRAM, ROCm 7.x, current master @ f53577… llama-cpp pin) with 4 preloaded models pushing usage to ~92 %:

   WARN [WatchDog] Memory usage exceeds threshold, evicting LRU backend type=GPU usage_percent=92.0 threshold_percent=90
   INFO [WatchDog] Memory reclaimer evicting LRU model model="Qwen2.5-72B-Instruct-GGUF"
   INFO [WatchDog] Memory reclaimer eviction complete model="Qwen2.5-72B-Instruct-GGUF"
   WARN Error freeing GPU resources error=rpc error: code = Unimplemented desc = model="Qwen2.5-72B-Instruct-GGUF"
   

   The reclaimer logs eviction complete but the llama-cpp gRPC backend's Free() returns Unimplemented, so VRAM is never actually released. Every 5 s tick the watchdog sees usage still >90 %, picks the next LRU victim, repeats. After all four preloaded models have been "evicted in name only", the next chat-completion that needs a freshly-loaded model crashes the backend on load:

   WARN Backend process exited unexpectedly id="qwen3.6-35b-a3b-apex" exitCode="-1"
   

   A few things that would make the feature actually safe to enable:

   1. Implement Free() in backend/cpp/llama-cpp/grpc-server.cpp so eviction physically releases the model + KV cache. This is the real fix.
   2. Until (1): in the reclaimer, when Free() returns Unimplemented, don't mark eviction as complete — fall back to the next-largest victim and add an exponential backoff before re-firing, otherwise the watchdog spins for nothing while load operations queue up behind it.
   3. The new feat: Log backend exit code (feat: Log backend exit code #9581, just merged) will help here — combined with surfacing the Unimplemented warning to the UI/logs as an error rather than warn would let users immediately see the misconfiguration.

   Workaround for anyone hitting this on llama-cpp today: set memory_reclaimer_enabled: false in runtime_settings.json, restart, pin the models you need via LOCALAI_LOAD_TO_MEMORY. Tracked separately in our memory as a regression to watch on every upstream sync.
5. SuperMarioYL

   added a commit that references this issue on May 3, 2026

   feat(watchdog): add size-aware LRU eviction mode

   ...

   This issue will close once commit 6077f91 is merged into the 'master' branch.

   6077f91
6. mudler

   closed this as completedin #9527on Jun 21, 2026
7. pull

   added a commit that references this issue on Jun 21, 2026

   feat(watchdog): add size-aware LRU eviction mode (mudler#9527)

   ...

   This issue will close once commit b50b1fe is merged into the 'master' branch.

   b50b1fe