---
name: LLM model manager
type: architecture-map-page
slice: 04-llm-formatting-mcp
last_verified: 2026/05/09
---

# LLM model manager

> **Where you are:** [Architecture map](../README.md) → [LLM, Formatting, MCP](README.md) → Model manager

**Plain English summary.** The four-tier Qwen registry, on-disk paths, resumable HTTP download, SHA-256 verification, hardware-tier recommendation, and on-disk delete. Single source of truth for every model fact: file name, size, expected SHA, Hugging Face URL, RAM and VRAM minimums, and human-friendly description.

## At a glance

- Crate: `lumotia-llm`
- Path: `crates/llm/src/model_manager.rs`
- LOC: 486
- Public surface:
  - `pub enum LlmModelId { Qwen3_5_2B_Q4, Qwen3_5_4B_Q4, Qwen3_5_9B_Q4, Qwen3_6_27B_Q4 }` (`crates/llm/src/model_manager.rs:14`)
  - `LlmModelId::default_tier`, `as_str`, `display_name`, `file_name`, `size_bytes`, `minimum_ram_bytes`, `recommended_vram_bytes`, `description`, `hf_url`, `sha256` (one per tier)
  - `impl Display for LlmModelId`, `impl FromStr for LlmModelId`
  - `pub struct LlmModelInfo` (`:152`) — serde-camelCase serialised summary for the frontend
  - `pub enum DownloadError` (`:164`)
  - `pub fn all_models() -> &'static [LlmModelId]` (`:213`)
  - `pub fn model_info(id: LlmModelId) -> LlmModelInfo` (`:217`)
  - `pub fn recommend_tier(total_ram_bytes: u64, total_vram_bytes: Option<u64>) -> LlmModelId` (`:229`)
  - `pub fn model_dir() -> PathBuf` (`:242`)
  - `pub fn model_path(id: LlmModelId) -> PathBuf` (`:246`)
  - `pub fn partial_download_path(id: LlmModelId) -> PathBuf` (`:250`)
  - `pub fn is_downloaded(id: LlmModelId) -> bool` (`:254`)
  - `pub fn delete_model(id: LlmModelId) -> io::Result<()>` (`:258`)
  - `pub async fn download_model<F>(id: LlmModelId, on_progress: F) -> Result<(), DownloadError>` where `F: FnMut(u64, u64) + Send + 'static` (`:272`)
- External deps that matter: `reqwest` with `rustls-tls` and `stream` features (no native TLS, no compress), `sha2`, `tokio` for async file IO, `futures-util::StreamExt`, `lumotia-core` for `paths::app_paths()`.
- Tauri command that calls this (slice 2, best guess): `commands::models::download_model` (`src-tauri/src/commands/models.rs:516`), wired via `src-tauri/src/lib.rs:325`. Plus `commands::llm::*` calls `model_manager::recommend_tier` at `src-tauri/src/commands/llm.rs:35` and `model_manager::download_model` at `src-tauri/src/commands/llm.rs:70`.

## What's in here

### The four tiers

Each entry in `LlmModelId` carries:

| Tier | File name | Size | Min RAM | Rec VRAM | Description |
|---|---|---|---|---|---|
| `Qwen3_5_2B_Q4` | `Qwen3.5-2B-Q4_K_M.gguf` | ~1.28 GB | 8 GiB | n/a (CPU) | Minimal tier for 8 GB RAM and CPU-heavy machines. |
| `Qwen3_5_4B_Q4` | `Qwen3.5-4B-Q4_K_M.gguf` | ~2.74 GB | 16 GiB | 6 GiB | Standard tier for cleanup and task extraction on 16 GB systems. (default tier) |
| `Qwen3_5_9B_Q4` | `Qwen3.5-9B-Q4_K_M.gguf` | ~5.68 GB | 32 GiB | 12 GiB | High tier for 32 GB RAM with a 12 GB+ GPU. |
| `Qwen3_6_27B_Q4` | `Qwen3.6-27B-Q4_K_M.gguf` | ~16.82 GB | 64 GiB | 24 GiB | Maximum tier for 64 GB RAM with a 24 GB GPU; partial CPU offload below that. |

All Q4_K_M GGUF, all from `unsloth/Qwen3.5-*-GGUF` and `unsloth/Qwen3.6-*-GGUF` HF repos. URLs pin a specific revision hash so re-uploads do not silently change the file behind us. SHA-256 values pin the exact bytes.

### `default_tier` (`crates/llm/src/model_manager.rs:26`)

Returns `Qwen3_5_4B_Q4`. Used by `LlmEngine::load(&Path)` when no tier is specified.

### `recommend_tier(total_ram_bytes, total_vram_bytes)` (`crates/llm/src/model_manager.rs:229`)

Tier selection logic, ordered most-capable first:

1. `vram >= 24 GiB && ram >= 64 GiB` → `Qwen3_6_27B_Q4`
2. `vram >= 12 GiB && ram >= 32 GiB` → `Qwen3_5_9B_Q4`
3. `vram >= 6 GiB || ram >= 16 GiB` → `Qwen3_5_4B_Q4`
4. otherwise → `Qwen3_5_2B_Q4`

`vram` defaults to 0 when the option is `None`, so a CPU-only machine takes the OR branch on the third rule. Test at `crates/llm/src/model_manager.rs:418` asserts a 16 GiB RAM machine with no GPU gets the 4B tier.

### Path helpers (`crates/llm/src/model_manager.rs:242-256`)

- `model_dir()` → `lumotia_core::paths::app_paths().llm_models_dir()`. The on-disk root, owned by `lumotia-core` (slice 5).
- `model_path(id)` → `model_dir().join(id.file_name())`. The final destination once a download completes.
- `partial_download_path(id)` → `model_path(id).with_extension("gguf.part")`. Where in-flight downloads accumulate.
- `is_downloaded(id)` → `model_path(id).exists()`. Cheap check; does not validate SHA.
- `delete_model(id)` → removes both `model_path` and `partial_download_path` (each only if present). Sync — runs on a regular thread.

### `download_model` and `download_impl` (`crates/llm/src/model_manager.rs:272`, `:307`)

The flagship entry point. Steps:

1. **Acquire a `DownloadReservation`.** A process-global `LazyLock<Mutex<HashSet<LlmModelId>>>` (`:183`) holds the set of in-flight downloads. The reservation's `Drop` releases the slot. A second concurrent download for the same tier returns `DownloadError::Http("download already in progress for {tier}")`. Different tiers can download in parallel.
2. **`tokio::fs::create_dir_all(model_dir())`.** Idempotent. Surfaces IO errors.
3. **If `dest` already exists, verify SHA.** A re-download of an already-correct file short-circuits to `Ok(())`. A SHA mismatch deletes the file and falls through to a fresh download. This is what makes `download_model` safe to call from a frontend that does not know whether the file is present.
4. **Call `download_impl(url, expected_sha, dest, on_progress)`.**

`download_impl` internals:

- **Resume detection.** `resume_from = tokio::fs::metadata(&tmp).await.ok().map(|m| m.len()).unwrap_or(0)`. If a `.gguf.part` file exists, we resume from its length.
- **`reqwest` client** with a 30-second connect timeout, `lumotia/0.1.0` user-agent, no aggressive compression (`stream` feature).
- **`Range: bytes={resume_from}-` header** when `resume_from > 0`. If the server responds with anything other than 206 PARTIAL_CONTENT to a ranged request, we return `DownloadError::ResumeUnsupported` rather than starting over silently.
- **Total-size resolution.** For a 200 OK response, `Content-Length` is the total. For a 206, parse `Content-Range: bytes start-end/total` to recover the underlying size; fall back to `content_length() + resume_from` if the header is missing.
- **Hasher pre-feed.** When resuming, the existing `.gguf.part` content is read once and fed into the SHA hasher so the final hash covers the entire file, not just the new chunks.
- **Append-mode write.** The `.gguf.part` is opened with `create + append` so a resumed write naturally lands at the end.
- **Progress callback.** `on_progress(downloaded, total)` is called once per chunk. The user-supplied closure typically forwards to a Tauri event for the frontend progress bar.
- **SHA verification.** After the stream ends, `format!("{:x}", hasher.finalize())` is compared against the expected hex string. A mismatch deletes the partial file and returns `DownloadError::ShaMismatch { expected, actual }`. There is no retry — the caller decides whether to re-attempt.
- **Atomic rename.** `tokio::fs::rename(&tmp, dest)` is the final step. Only after SHA passes does the file appear at its real path.

### `DownloadError` (`crates/llm/src/model_manager.rs:164`)

Variants:

- `Http(String)` — anything from "DNS failed" to "server replied 500".
- `Io(io::Error)` — `#[from]` so `tokio::fs` errors lift cleanly.
- `ShaMismatch { expected: String, actual: String }` — caller-actionable.
- `ResumeUnsupported` — the server does not support range requests; rare but possible if HF or a CDN changes behaviour.

### `LlmModelInfo` (`crates/llm/src/model_manager.rs:152`)

Serde camelCase struct mirroring the per-tier metadata for frontend consumption. Every field is `&'static str` or `u64` so cloning is cheap. `model_info(id)` builds one on demand.

## Data flow

Download path:

```
Tauri command (commands::models::download_model)
  → lumotia_llm::model_manager::download_model(id, on_progress)
       → DownloadReservation::acquire(id)  (Http error if duplicate)
       → tokio::fs::create_dir_all(model_dir())
       → if dest.exists(): sha256_file(dest); short-circuit if match, else delete
       → download_impl(hf_url, expected_sha, dest, on_progress)
            → resume_from = part-file size (if exists)
            → reqwest GET with Range header (resume) or plain (fresh)
            → hasher pre-feed from existing partial file
            → stream chunks → write_all → hasher.update → on_progress
            → SHA finalise + compare
            → tokio::fs::rename(part → final)
       → DownloadReservation drops, releases slot
  → returns Ok(()) or DownloadError
```

Tier-recommend path:

```
sysinfo or lumotia_core::system → (ram_bytes, Option<vram_bytes>)
  → recommend_tier(ram, vram) → LlmModelId
  → load_model(id, model_path(id), use_gpu)
```

## Watch-outs

- **`DownloadReservation` is process-local.** A user running two Lumotia processes against the same on-disk directory could race. We do not file-lock the `.gguf.part`. Realistically rare — but if it ever happens, the SHA check catches a corrupt result.
- **No retry, no exponential backoff.** A flaky network surfaces as `DownloadError::Http` and the frontend has to ask the user to retry. Resume keeps that retry cheap (only the missing tail re-fetches).
- **HF revision pinning is manual.** `hf_url()` includes the commit hash. Updating to a new upstream revision means changing the URL and the SHA in lockstep. No tooling enforces that the SHA still matches the URL — manual verification at upgrade time.
- **`size_bytes` is informational, not enforced.** It is shown in the UI before download starts. The download trusts `Content-Length` (or computed from `Content-Range`) for actual progress, so a server that lies about size shows a misleading bar but still verifies SHA at the end.
- **`minimum_ram_bytes` is advisory.** Nothing in the loader checks RAM at runtime. A user with 4 GB of RAM picking the 9B tier will see a llama.cpp OOM. The frontend should gate selection on this number.
- **`recommended_vram_bytes: None` for the 2B tier means "no GPU recommended", not "GPU optional".** The 2B tier is the CPU-only path; the others can run partially-offloaded but get warnings from llama.cpp.
- **`is_downloaded` does not verify SHA.** It only checks file existence. A corrupted file passes. The next `download_model` call would catch it, but a `LlmEngine::load_model` would fail with a llama.cpp parse error first. Worth a follow-up to expose a `verify_model(id) -> bool` if user-facing "model integrity check" UX surfaces.

## See also

- [LLM engine](llm-engine.md) — load path and what consumes the file
- [LLM Cargo features](llm-cargo-features.md)
- [Slice README — model registry drift entry](README.md)