Lumotia/docs/architecture-map/04-llm-formatting-mcp/llm-model-manager.md

name, type, slice, last_verified

name	type	slice	last_verified
LLM model manager	architecture-map-page	04-llm-formatting-mcp	2026/05/09

LLM model manager

Where you are: Architecture map → LLM, Formatting, MCP → Model manager

Plain English summary. The four-tier Qwen registry, on-disk paths, resumable HTTP download, SHA-256 verification, hardware-tier recommendation, and on-disk delete. Single source of truth for every model fact: file name, size, expected SHA, Hugging Face URL, RAM and VRAM minimums, and human-friendly description.

At a glance

• Crate: lumotia-llm
• Path: crates/llm/src/model_manager.rs
• LOC: 486
• Public surface:
  • pub enum LlmModelId { Qwen3_5_2B_Q4, Qwen3_5_4B_Q4, Qwen3_5_9B_Q4, Qwen3_6_27B_Q4 } (crates/llm/src/model_manager.rs:14)
  • LlmModelId::default_tier, as_str, display_name, file_name, size_bytes, minimum_ram_bytes, recommended_vram_bytes, description, hf_url, sha256 (one per tier)
  • impl Display for LlmModelId, impl FromStr for LlmModelId
  • pub struct LlmModelInfo (:152) — serde-camelCase serialised summary for the frontend
  • pub enum DownloadError (:164)
  • pub fn all_models() -> &'static [LlmModelId] (:213)
  • pub fn model_info(id: LlmModelId) -> LlmModelInfo (:217)
  • pub fn recommend_tier(total_ram_bytes: u64, total_vram_bytes: Option<u64>) -> LlmModelId (:229)
  • pub fn model_dir() -> PathBuf (:242)
  • pub fn model_path(id: LlmModelId) -> PathBuf (:246)
  • pub fn partial_download_path(id: LlmModelId) -> PathBuf (:250)
  • pub fn is_downloaded(id: LlmModelId) -> bool (:254)
  • pub fn delete_model(id: LlmModelId) -> io::Result<()> (:258)
  • pub async fn download_model<F>(id: LlmModelId, on_progress: F) -> Result<(), DownloadError> where F: FnMut(u64, u64) + Send + 'static (:272)
• External deps that matter: reqwest with rustls-tls and stream features (no native TLS, no compress), sha2, tokio for async file IO, futures-util::StreamExt, lumotia-core for paths::app_paths().
• Tauri command that calls this (slice 2, best guess): commands::models::download_model (src-tauri/src/commands/models.rs:516), wired via src-tauri/src/lib.rs:325. Plus commands::llm::* calls model_manager::recommend_tier at src-tauri/src/commands/llm.rs:35 and model_manager::download_model at src-tauri/src/commands/llm.rs:70.

What's in here

The four tiers

Each entry in LlmModelId carries:

Tier	File name	Size	Min RAM	Rec VRAM	Description
Qwen3_5_2B_Q4	Qwen3.5-2B-Q4_K_M.gguf	~1.28 GB	8 GiB	n/a (CPU)	Minimal tier for 8 GB RAM and CPU-heavy machines.
Qwen3_5_4B_Q4	Qwen3.5-4B-Q4_K_M.gguf	~2.74 GB	16 GiB	6 GiB	Standard tier for cleanup and task extraction on 16 GB systems. (default tier)
Qwen3_5_9B_Q4	Qwen3.5-9B-Q4_K_M.gguf	~5.68 GB	32 GiB	12 GiB	High tier for 32 GB RAM with a 12 GB+ GPU.
Qwen3_6_27B_Q4	Qwen3.6-27B-Q4_K_M.gguf	~16.82 GB	64 GiB	24 GiB	Maximum tier for 64 GB RAM with a 24 GB GPU; partial CPU offload below that.

All Q4_K_M GGUF, all from unsloth/Qwen3.5-*-GGUF and unsloth/Qwen3.6-*-GGUF HF repos. URLs pin a specific revision hash so re-uploads do not silently change the file behind us. SHA-256 values pin the exact bytes.

default_tier (crates/llm/src/model_manager.rs:26)

Returns Qwen3_5_4B_Q4. Used by LlmEngine::load(&Path) when no tier is specified.

recommend_tier(total_ram_bytes, total_vram_bytes) (crates/llm/src/model_manager.rs:229)

Tier selection logic, ordered most-capable first:

1. vram >= 24 GiB && ram >= 64 GiB → Qwen3_6_27B_Q4
2. vram >= 12 GiB && ram >= 32 GiB → Qwen3_5_9B_Q4
3. vram >= 6 GiB || ram >= 16 GiB → Qwen3_5_4B_Q4
4. otherwise → Qwen3_5_2B_Q4

vram defaults to 0 when the option is None, so a CPU-only machine takes the OR branch on the third rule. Test at crates/llm/src/model_manager.rs:418 asserts a 16 GiB RAM machine with no GPU gets the 4B tier.

Path helpers (crates/llm/src/model_manager.rs:242-256)

• model_dir() → lumotia_core::paths::app_paths().llm_models_dir(). The on-disk root, owned by lumotia-core (slice 5).
• model_path(id) → model_dir().join(id.file_name()). The final destination once a download completes.
• partial_download_path(id) → model_path(id).with_extension("gguf.part"). Where in-flight downloads accumulate.
• is_downloaded(id) → model_path(id).exists(). Cheap check; does not validate SHA.
• delete_model(id) → removes both model_path and partial_download_path (each only if present). Sync — runs on a regular thread.

download_model and download_impl (crates/llm/src/model_manager.rs:272, :307)

The flagship entry point. Steps:

1. Acquire a DownloadReservation. A process-global LazyLock<Mutex<HashSet<LlmModelId>>> (:183) holds the set of in-flight downloads. The reservation's Drop releases the slot. A second concurrent download for the same tier returns DownloadError::Http("download already in progress for {tier}"). Different tiers can download in parallel.
2. tokio::fs::create_dir_all(model_dir()). Idempotent. Surfaces IO errors.
3. If dest already exists, verify SHA. A re-download of an already-correct file short-circuits to Ok(()). A SHA mismatch deletes the file and falls through to a fresh download. This is what makes download_model safe to call from a frontend that does not know whether the file is present.
4. Call download_impl(url, expected_sha, dest, on_progress).

download_impl internals:

• Resume detection. resume_from = tokio::fs::metadata(&tmp).await.ok().map(|m| m.len()).unwrap_or(0). If a .gguf.part file exists, we resume from its length.
• reqwest client with a 30-second connect timeout, lumotia/0.1.0 user-agent, no aggressive compression (stream feature).
• Range: bytes={resume_from}- header when resume_from > 0. If the server responds with anything other than 206 PARTIAL_CONTENT to a ranged request, we return DownloadError::ResumeUnsupported rather than starting over silently.
• Total-size resolution. For a 200 OK response, Content-Length is the total. For a 206, parse Content-Range: bytes start-end/total to recover the underlying size; fall back to content_length() + resume_from if the header is missing.
• Hasher pre-feed. When resuming, the existing .gguf.part content is read once and fed into the SHA hasher so the final hash covers the entire file, not just the new chunks.
• Append-mode write. The .gguf.part is opened with create + append so a resumed write naturally lands at the end.
• Progress callback. on_progress(downloaded, total) is called once per chunk. The user-supplied closure typically forwards to a Tauri event for the frontend progress bar.
• SHA verification. After the stream ends, format!("{:x}", hasher.finalize()) is compared against the expected hex string. A mismatch deletes the partial file and returns DownloadError::ShaMismatch { expected, actual }. There is no retry — the caller decides whether to re-attempt.
• Atomic rename. tokio::fs::rename(&tmp, dest) is the final step. Only after SHA passes does the file appear at its real path.

DownloadError (crates/llm/src/model_manager.rs:164)

Variants:

• Http(String) — anything from "DNS failed" to "server replied 500".
• Io(io::Error) — #[from] so tokio::fs errors lift cleanly.
• ShaMismatch { expected: String, actual: String } — caller-actionable.
• ResumeUnsupported — the server does not support range requests; rare but possible if HF or a CDN changes behaviour.

LlmModelInfo (crates/llm/src/model_manager.rs:152)

Serde camelCase struct mirroring the per-tier metadata for frontend consumption. Every field is &'static str or u64 so cloning is cheap. model_info(id) builds one on demand.

Data flow

Download path:

Tauri command (commands::models::download_model)
  → lumotia_llm::model_manager::download_model(id, on_progress)
       → DownloadReservation::acquire(id)  (Http error if duplicate)
       → tokio::fs::create_dir_all(model_dir())
       → if dest.exists(): sha256_file(dest); short-circuit if match, else delete
       → download_impl(hf_url, expected_sha, dest, on_progress)
            → resume_from = part-file size (if exists)
            → reqwest GET with Range header (resume) or plain (fresh)
            → hasher pre-feed from existing partial file
            → stream chunks → write_all → hasher.update → on_progress
            → SHA finalise + compare
            → tokio::fs::rename(part → final)
       → DownloadReservation drops, releases slot
  → returns Ok(()) or DownloadError

Tier-recommend path:

sysinfo or lumotia_core::system → (ram_bytes, Option<vram_bytes>)
  → recommend_tier(ram, vram) → LlmModelId
  → load_model(id, model_path(id), use_gpu)

Watch-outs

• DownloadReservation is process-local. A user running two Lumotia processes against the same on-disk directory could race. We do not file-lock the .gguf.part. Realistically rare — but if it ever happens, the SHA check catches a corrupt result.
• No retry, no exponential backoff. A flaky network surfaces as DownloadError::Http and the frontend has to ask the user to retry. Resume keeps that retry cheap (only the missing tail re-fetches).
• HF revision pinning is manual. hf_url() includes the commit hash. Updating to a new upstream revision means changing the URL and the SHA in lockstep. No tooling enforces that the SHA still matches the URL — manual verification at upgrade time.
• size_bytes is informational, not enforced. It is shown in the UI before download starts. The download trusts Content-Length (or computed from Content-Range) for actual progress, so a server that lies about size shows a misleading bar but still verifies SHA at the end.
• minimum_ram_bytes is advisory. Nothing in the loader checks RAM at runtime. A user with 4 GB of RAM picking the 9B tier will see a llama.cpp OOM. The frontend should gate selection on this number.
• recommended_vram_bytes: None for the 2B tier means "no GPU recommended", not "GPU optional". The 2B tier is the CPU-only path; the others can run partially-offloaded but get warnings from llama.cpp.
• is_downloaded does not verify SHA. It only checks file existence. A corrupted file passes. The next download_model call would catch it, but a LlmEngine::load_model would fail with a llama.cpp parse error first. Worth a follow-up to expose a verify_model(id) -> bool if user-facing "model integrity check" UX surfaces.

See also

• LLM engine — load path and what consumes the file
• LLM Cargo features
• Slice README — model registry drift entry