---
name: LLM engine and llama-cpp-2 lifecycle
type: architecture-map-page
slice: 04-llm-formatting-mcp
last_verified: 2026/05/09
---

# LLM engine and llama-cpp-2 lifecycle

> **Where you are:** [Architecture map](../README.md) → [LLM, Formatting, MCP](README.md) → LLM engine

**Plain English summary.** `LlmEngine` is the cloneable handle every part of Lumotia uses to talk to a local Qwen model. It owns the llama-cpp-2 backend, holds a single loaded model behind a mutex, and exposes one low-level `generate` plus four high-level surfaces. Everything else in the LLM crate (prompts, grammars, model manager) feeds into this type.

## At a glance

- Crate: `lumotia-llm`
- Path: `crates/llm/src/lib.rs`
- LOC: 570
- Public surface (from `lib.rs`):
  - `pub struct LlmEngine` (`crates/llm/src/lib.rs:84`) with `pub fn new`, `pub fn load`, `pub fn load_model`, `pub fn unload`, `pub fn is_loaded`, `pub fn loaded_model`, `pub fn loaded_model_id`, `pub fn generate`, `pub fn cleanup_text`, `pub fn decompose_task`, `pub fn decompose_task_with_feedback`, `pub fn extract_tasks`, `pub fn extract_tasks_with_feedback`, `pub fn extract_content_tags`.
  - `pub enum EngineError` (`crates/llm/src/lib.rs:29`) with variants `NotLoaded`, `LoadFailed`, `PromptTooLong`, `Inference`, `InvalidJson`.
  - `pub struct GenerationConfig` (`crates/llm/src/lib.rs:50`).
  - `pub struct LoadedModelState` (`crates/llm/src/lib.rs:70`).
  - Re-exports: `CONTENT_TAGS_GRAMMAR`, `recommend_tier`, `LlmModelId`, `LlmModelInfo`, `is_valid_intent`, `ContentTags`, `CONTENT_TAGS_SYSTEM`, `INTENT_CLOSED_SET`.
- External deps that matter: `llama-cpp-2 = 0.1.144`, `tokio` (only for the async download path in `model_manager`; `generate` itself is sync), `serde`, `tracing`, `lumotia-core` for thread tuning.
- Tauri command that calls this (slice 2, best guess):
  - Lifecycle: `llm_load_model`, `llm_unload_model`, `llm_load_recommended_model`, `llm_is_loaded` etc. defined in `src-tauri/src/commands/llm.rs`. Observed call sites at `src-tauri/src/commands/llm.rs:116` (`engine.load_model`), `:128` (`engine.unload`), `:147` (`engine.is_loaded`), `:244` (`load_model` again from the recommended-tier path).

## What's in here

### `LlmEngine` struct (`crates/llm/src/lib.rs:84`)

A `Clone`-able newtype around `Arc<Mutex<LlmState>>`. Cloning shares the loaded model — every Tauri command holds its own clone but they all act on the same in-memory state. The state itself is a small `LlmState` struct (`:77`) with three options: the `LlamaBackend`, the `LlamaModel`, and the `LoadedModelState` metadata. Both backend and model are `Arc`s so a load while one inference is mid-flight does not free the model out from under the borrowed handle.

### `load` and `load_model` (`crates/llm/src/lib.rs:93`, `:97`)

`load` is a thin wrapper that pins the default tier (`LlmModelId::default_tier()`) and asks for GPU. The full `load_model` takes `(LlmModelId, &Path, use_gpu: bool)`.

Logic:

1. If a model with the same id, path, and GPU flag is already loaded, no-op return `Ok(())`. This makes repeated calls from the frontend cheap.
2. Reuse the cached `LlamaBackend` if present; otherwise initialise one. The backend is a process-global singleton inside llama-cpp-2; double-init is undefined-behaviour territory, so we keep ours alive across reloads.
3. `gpu_layers = if use_gpu { u32::MAX } else { 0 }`. There is no partial-offload path. Either every layer goes to the GPU or none do.
4. `LlamaModel::load_from_file(...)` with the path the model manager produced.

Failures collapse into `EngineError::LoadFailed(String)`.

### `unload` (`crates/llm/src/lib.rs:137`)

Drops the `Arc<LlamaModel>` and `Arc<LlamaBackend>`, clears `loaded`. The Tauri layer calls this from `llm_unload_model` and indirectly via `llm_delete_model` when the active model is the one being deleted.

### `is_loaded`, `loaded_model`, `loaded_model_id` (`crates/llm/src/lib.rs:145`, `:149`, `:153`)

Cheap getters. `loaded_model` returns `Option<LoadedModelState>` so the frontend can show which tier is active.

### `generate` (`crates/llm/src/lib.rs:157`)

The sync, low-level inference primitive. Steps:

1. **Tokenise the prompt.** `model.str_to_token(prompt, AddBos::Never)` — the `AddBos::Never` matters: `render_chat_prompt` already emits the BOS token via the Qwen chat template.
2. **Empty-prompt short-circuit.** If tokenisation produced zero tokens, return `Ok(String::new())` without touching the GPU.
3. **Preflight context window.** `preflight_context_window(prompt_tokens.len(), config.max_tokens)` (`:436`) errors with `EngineError::PromptTooLong { ... }` when `prompt_tokens + max_tokens + 64 reserve` exceeds the 8192 cap. Fixed sizing — see the watch-out about MAX_CONTEXT_TOKENS below. This was RB-10 from the 2026-04-22 review (`docs/issues/llm-prompt-preflight.md`).
4. **Compute thread count.** `gpu_offloaded = use_gpu && gpu_layers >= model.n_layer()`. The compiler can prove this is trivially true today because `gpu_layers` is `u32::MAX` whenever `use_gpu` is set. The redundant check is documented inline (`:169-173`) as a placeholder for future per-layer residency parsing of llama.cpp's log output. `inference_thread_count(Workload::Llm, gpu_offloaded)` from `lumotia_core::tuning` returns the physical core count adjusted for the workload class.
5. **Build context params.** `n_ctx` from preflight, `n_batch` and `n_ubatch` clamped to `[max(prompt_tokens, 512), n_ctx]`, `n_threads` and `n_threads_batch` both set to the computed thread count.
6. **Prefill.** A single `LlamaBatch` is built with every prompt token, the last token marked as the only logits-bearing position, then `ctx.decode(&mut batch)`.
7. **Sample loop.** A custom sampler chain (`build_sampler`, `:400`) is built from `config.grammar` (optional GBNF), `temperature`, and a fixed `GENERATION_SEED = 0`. For temperature 0.0 (the only value the high-level surfaces use) we attach `LlamaSampler::greedy()` after the optional grammar. For non-zero temperatures we attach `temp` then `dist` with the seed.
8. **Per-token loop.** Until either the model emits an EOG / EOS token, or `max_tokens` is hit, or a stop sequence appears in the running output:
   - Sample, then check `is_eog_token` and `token_eos` (Qwen's chat templates use both).
   - Detokenise the new token via a UTF-8 `encoding_rs` decoder so multi-byte sequences split across token boundaries do not garble.
   - Push to the running `String`, accept the token in the sampler.
   - Test for any of `config.stop_sequences` in the running buffer; truncate and break if one is found.
   - Otherwise re-batch the new token alone and `ctx.decode` it for the next round.
9. **Trim and return.** `Ok(generated.trim().to_string())`.

### High-level surfaces (also on `LlmEngine`)

Each is documented in its own page:

- [`cleanup_text`](llm-cleanup-text.md) — `crates/llm/src/lib.rs:232`
- [`decompose_task`](llm-decompose-task.md) and `decompose_task_with_feedback` — `:254`, `:267`
- [`extract_tasks`](llm-extract-tasks.md) and `extract_tasks_with_feedback` — `:294`, `:358`
- [`extract_content_tags`](llm-extract-content-tags.md) — `:306`

All four set `temperature: 0.0` and pass at least one llama-cpp stop sequence (`<|im_end|>` and `<|im_end_of_text|>`). All four call `render_chat_prompt` to apply the model's tokenizer-bundled chat template, with a ChatML fallback.

### `render_chat_prompt` (`crates/llm/src/lib.rs:462`)

Tries `model.chat_template(None)`. If the GGUF carries a tokenizer-defined template, that is used. Otherwise we log a `tracing::warn!` and fall back to a built-in `LlamaChatTemplate::new("chatml")`. ChatML is what the Qwen3.5 / 3.6 family expects, so the fallback is safe in practice; the warn is there for the day someone tries a model from outside the registered family.

### `parse_string_array` (`crates/llm/src/lib.rs:489`)

Helper that the array-returning surfaces share. Calls `serde_json::from_str::<Vec<String>>(raw.trim())`, then trims items, drops empties, and dedupes case-insensitively while preserving first-seen ordering. The case-insensitive dedupe matters: the LLM occasionally emits `"Buy milk"` and `"buy milk"` in the same array, and the user does not want to see both.

### `build_sampler` (`crates/llm/src/lib.rs:400`)

Composes `LlamaSampler::grammar(model, grammar, "root")` (when `config.grammar` is set), then either `LlamaSampler::greedy()` for temperature 0.0 or `LlamaSampler::temp(temperature)` plus `LlamaSampler::dist(GENERATION_SEED)` for non-zero temperatures. Single-element chains are returned as-is; multi-element chains use `LlamaSampler::chain_simple`.

### Internal constants (`crates/llm/src/lib.rs:23-26`)

- `DEFAULT_CONTEXT_TOKENS = 4096` — minimum context window we ever allocate.
- `MAX_CONTEXT_TOKENS = 8192` — hard cap. Both the preflight error and the realised `n_ctx` are bounded here.
- `CONTEXT_RESERVE_TOKENS = 64` — extra headroom subtracted from the prompt budget so a tight fit never wedges the sampler.
- `GENERATION_SEED = 0` — fixed sampling seed. Combined with temperature 0.0, makes greedy decoding fully deterministic for the high-level surfaces.

## Data flow

```
caller prompt + GenerationConfig
  → str_to_token (AddBos::Never)
  → preflight_context_window  (or PromptTooLong error)
  → tuning::inference_thread_count (with gpu_offloaded)
  → LlamaContextParams (n_ctx, n_batch, n_ubatch, n_threads)
  → LlamaModel::new_context
  → LlamaBatch (prefill, last token logits)
  → ctx.decode
  → loop:
       LlamaSampler::sample (greedy or temp+dist, optional grammar)
       → is_eog_token / token_eos check
       → token_to_piece (UTF-8 incremental decoder)
       → stop-sequence check
       → batch.clear + add(next, cursor) + ctx.decode
  → trim + return String
```

The high-level surfaces wrap this with: a chat-template render in front, and (for array surfaces) `parse_string_array` plus a typed JSON deserialise behind.

## Prompts and grammars

This file does not hold prompts or grammars itself. See [`llm-prompts-and-grammars.md`](llm-prompts-and-grammars.md) for the catalogue. The engine consumes them by reference in each surface.

## Watch-outs

- **Mutex-protected single model.** `LlmEngine` allows only one model loaded at a time. Two concurrent `generate` calls serialise on the underlying llama context (each call builds its own `new_context` from the shared `LlamaModel`, so the model weights are shared but the KV cache is per-call). The Tauri layer wraps each high-level call in `tokio::task::spawn_blocking` because `generate` is sync and blocks the executor for hundreds of milliseconds at minimum.
- **`MAX_CONTEXT_TOKENS = 8192` is a process-wide cap** regardless of which tier is loaded. The 27B tier's native context is much larger; we are deliberately leaving headroom on the table to keep the preflight predictable. Surfaced in the slice README's debt section.
- **`u32::MAX` GPU offload.** The engine has no concept of partial offload. On a low-VRAM machine that cannot fit all layers, llama.cpp will emit warnings and fall back to mixed CPU/GPU automatically, but our `gpu_offloaded` boolean tells `inference_thread_count` we are fully GPU-resident. When this matters (battery, throttling), the consumer is `lumotia_core::tuning` — it picks a bigger thread count when it thinks the CPU is idle. Trivial-true today; tracked as observability gap (commit `052265b`).
- **GBNF parser quirks.** llama-cpp-2's GBNF is strict about whitespace. Each grammar in [`llm-prompts-and-grammars.md`](llm-prompts-and-grammars.md) carries an explicit `ws` rule and `r#""#` raw strings — refactors that try to "tidy" the grammar literal by stripping the trailing newline have, in the past, broken `LlamaSampler::grammar` with cryptic parse errors.
- **Stop sequences are post-detokenisation substring matches.** They run on the running `String`, not on token ids. A multi-byte stop string that splits across a token boundary still matches because the UTF-8 decoder buffers partial bytes. A stop string that contains characters the chat template re-emits as part of normal output (e.g. a literal `<|`) will trigger early termination — only use the EOG sentinels we already use.
- **Chat template fallback to ChatML.** If `model.chat_template(None)` errors, we warn-log and use `LlamaChatTemplate::new("chatml")`. The warn-log fires once per `generate` call, not once per session — keep an eye on log volume if a non-Qwen model is ever loaded.
- **Backend reuse across loads.** `LlmState.backend` is intentionally not dropped on `unload`. Re-init of `LlamaBackend` after a previous init is unsupported by llama-cpp-2; we keep one alive for the lifetime of the process.

## See also

- [LLM cleanup_text](llm-cleanup-text.md)
- [LLM decompose_task](llm-decompose-task.md)
- [LLM extract_tasks](llm-extract-tasks.md)
- [LLM extract_content_tags](llm-extract-content-tags.md)
- [Prompts and grammars catalogue](llm-prompts-and-grammars.md)
- [Model manager](llm-model-manager.md)
- [Cargo features](llm-cargo-features.md)
- [Tests](llm-tests.md)
- [Slice README](README.md)