Race-3 (conf 86): `LlmEngine::load_model` previously held the inner
`std::sync::Mutex` for the entire `LlamaBackend::init` +
`LlamaModel::load_from_file` call (5-15 s on a cold load). `is_loaded()`,
`loaded_model()`, and `loaded_model_id()` all take that same mutex and
are called from sync Tauri handlers (`get_llm_status`, `check_llm_model`,
`delete_llm_model`, `test_llm_model`) WITHOUT `spawn_blocking`. During a
first-run load, parallel `refreshLlmStatus()` polls from the frontend
parked tokio worker threads on the std-mutex; a handful of concurrent
status polls was enough to deadlock the default `num_cpus`-sized Tauri
runtime.
Lifecycle-1 (conf 80): same function held the OLD `Arc<LlamaModel>` in
`guard.model` during the new `load_from_file` call, so a model swap
peaked at ~2x VRAM. A 27B Q4 (~17 GB) swap OOMed a 24 GB card even
though either model fit alone.
Fix: redesign `load_model` so the slow llama-cpp work happens OUTSIDE
the mutex.
1. Short crit section: compare against currently-loaded triple —
return Ok on match (no-op fast path).
2. CAS a new `loading: AtomicBool` from false → true. If a load is
already in flight, refuse with the new `EngineError::AlreadyLoading`
rather than starting a parallel one. A `LoadingGuard` RAII drop
clears the flag on every exit path including panic.
3. Short crit section: drop the OLD model Arc (releasing VRAM via
`llama_free_model`) BEFORE the new load begins — fixes Lifecycle-1.
4. Heavy `LlamaBackend::init` + `LlamaModel::load_from_file` run
OUTSIDE the mutex.
5. Short crit section: install the new state.
`is_loaded()`, `loaded_model()`, `loaded_model_id()` now only contend
on the brief state-mutation sections, not the multi-second file load.
A new `is_loading()` accessor exposes the in-flight state for callers
that need to distinguish "loading" from "not loaded".
Backend lifecycle: `LlamaBackend::init` is process-singleton —
llama-cpp-2 enforces this via `LLAMA_BACKEND_INITIALIZED: AtomicBool`
and returns `BackendAlreadyInitialized` on a second call. The Drop impl
DOES call `llama_backend_free` and flips the flag back, so re-init
would technically work, but we keep the backend Arc resident across
loads/unloads to avoid init/free churn. The Lifecycle-1 fix drops only
the model Arc, NOT the backend Arc — dropping the backend mid-swap
would still work (the new load would re-init), but the comment trail
documents the singleton contract for the next reader.
`is_loaded()` now reports false briefly during a swap window (model
dropped, new model not yet installed). Documented in the doc comment.
Callers wanting "model X is loaded" must check `loaded_model_id()`
against their target, not just `is_loaded()`.
Regression tests added in `crates/llm/src/lib.rs::tests`:
- `is_loaded_does_not_block_on_slow_load`: holds the lock-discipline
open via a Barrier and asserts `is_loaded()` / `loaded_model_id()`
return in ≤50 ms while the slow section is mid-flight. Verified
that a pre-fix structure (`std::sync::Mutex` held across a 500 ms
sleep) makes the probe block ~450 ms; the new structure makes it
return in microseconds.
- `second_concurrent_load_is_refused`: two concurrent
`__test_run_with_lock_discipline` calls; the second gets
`EngineError::AlreadyLoading` and never reaches its op closure.
Doubles as the TOCTOU guard for Race-4/5 at the engine layer.
A `pub(crate) #[cfg(test)] __test_run_with_lock_discipline` helper
exposes the locking skeleton (loading flag + drop-old-model + slow op
outside lock + install) without requiring a real GGUF on disk; this is
the harness the regression tests use.
Verification: cargo test --workspace + npm run check both green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Jake
cde985d0c1