Lumotia/docs/architecture-map/04-llm-formatting-mcp/llm-engine.md

name, type, slice, last_verified

name	type	slice	last_verified
LLM engine and llama-cpp-2 lifecycle	architecture-map-page	04-llm-formatting-mcp	2026/05/09

LLM engine and llama-cpp-2 lifecycle

Where you are: Architecture map → LLM, Formatting, MCP → LLM engine

Plain English summary. LlmEngine is the cloneable handle every part of Lumotia uses to talk to a local Qwen model. It owns the llama-cpp-2 backend, holds a single loaded model behind a mutex, and exposes one low-level generate plus four high-level surfaces. Everything else in the LLM crate (prompts, grammars, model manager) feeds into this type.

At a glance

• Crate: lumotia-llm
• Path: crates/llm/src/lib.rs
• LOC: 570
• Public surface (from lib.rs):
  • pub struct LlmEngine (crates/llm/src/lib.rs:84) with pub fn new, pub fn load, pub fn load_model, pub fn unload, pub fn is_loaded, pub fn loaded_model, pub fn loaded_model_id, pub fn generate, pub fn cleanup_text, pub fn decompose_task, pub fn decompose_task_with_feedback, pub fn extract_tasks, pub fn extract_tasks_with_feedback, pub fn extract_content_tags.
  • pub enum EngineError (crates/llm/src/lib.rs:29) with variants NotLoaded, LoadFailed, PromptTooLong, Inference, InvalidJson.
  • pub struct GenerationConfig (crates/llm/src/lib.rs:50).
  • pub struct LoadedModelState (crates/llm/src/lib.rs:70).
  • Re-exports: CONTENT_TAGS_GRAMMAR, recommend_tier, LlmModelId, LlmModelInfo, is_valid_intent, ContentTags, CONTENT_TAGS_SYSTEM, INTENT_CLOSED_SET.
• External deps that matter: llama-cpp-2 = 0.1.144, tokio (only for the async download path in model_manager; generate itself is sync), serde, tracing, lumotia-core for thread tuning.
• Tauri command that calls this (slice 2, best guess):
  • Lifecycle: llm_load_model, llm_unload_model, llm_load_recommended_model, llm_is_loaded etc. defined in src-tauri/src/commands/llm.rs. Observed call sites at src-tauri/src/commands/llm.rs:116 (engine.load_model), :128 (engine.unload), :147 (engine.is_loaded), :244 (load_model again from the recommended-tier path).

What's in here

LlmEngine struct (crates/llm/src/lib.rs:84)

A Clone-able newtype around Arc<Mutex<LlmState>>. Cloning shares the loaded model — every Tauri command holds its own clone but they all act on the same in-memory state. The state itself is a small LlmState struct (:77) with three options: the LlamaBackend, the LlamaModel, and the LoadedModelState metadata. Both backend and model are Arcs so a load while one inference is mid-flight does not free the model out from under the borrowed handle.

load and load_model (crates/llm/src/lib.rs:93, :97)

load is a thin wrapper that pins the default tier (LlmModelId::default_tier()) and asks for GPU. The full load_model takes (LlmModelId, &Path, use_gpu: bool).

Logic:

1. If a model with the same id, path, and GPU flag is already loaded, no-op return Ok(()). This makes repeated calls from the frontend cheap.
2. Reuse the cached LlamaBackend if present; otherwise initialise one. The backend is a process-global singleton inside llama-cpp-2; double-init is undefined-behaviour territory, so we keep ours alive across reloads.
3. gpu_layers = if use_gpu { u32::MAX } else { 0 }. There is no partial-offload path. Either every layer goes to the GPU or none do.
4. LlamaModel::load_from_file(...) with the path the model manager produced.

Failures collapse into EngineError::LoadFailed(String).

unload (crates/llm/src/lib.rs:137)

Drops the Arc<LlamaModel> and Arc<LlamaBackend>, clears loaded. The Tauri layer calls this from llm_unload_model and indirectly via llm_delete_model when the active model is the one being deleted.

is_loaded, loaded_model, loaded_model_id (crates/llm/src/lib.rs:145, :149, :153)

Cheap getters. loaded_model returns Option<LoadedModelState> so the frontend can show which tier is active.

generate (crates/llm/src/lib.rs:157)

The sync, low-level inference primitive. Steps:

1. Tokenise the prompt. model.str_to_token(prompt, AddBos::Never) — the AddBos::Never matters: render_chat_prompt already emits the BOS token via the Qwen chat template.
2. Empty-prompt short-circuit. If tokenisation produced zero tokens, return Ok(String::new()) without touching the GPU.
3. Preflight context window. preflight_context_window(prompt_tokens.len(), config.max_tokens) (:436) errors with EngineError::PromptTooLong { ... } when prompt_tokens + max_tokens + 64 reserve exceeds the 8192 cap. Fixed sizing — see the watch-out about MAX_CONTEXT_TOKENS below. This was RB-10 from the 2026-04-22 review (docs/issues/llm-prompt-preflight.md).
4. Compute thread count. gpu_offloaded = use_gpu && gpu_layers >= model.n_layer(). The compiler can prove this is trivially true today because gpu_layers is u32::MAX whenever use_gpu is set. The redundant check is documented inline (:169-173) as a placeholder for future per-layer residency parsing of llama.cpp's log output. inference_thread_count(Workload::Llm, gpu_offloaded) from lumotia_core::tuning returns the physical core count adjusted for the workload class.
5. Build context params. n_ctx from preflight, n_batch and n_ubatch clamped to [max(prompt_tokens, 512), n_ctx], n_threads and n_threads_batch both set to the computed thread count.
6. Prefill. A single LlamaBatch is built with every prompt token, the last token marked as the only logits-bearing position, then ctx.decode(&mut batch).
7. Sample loop. A custom sampler chain (build_sampler, :400) is built from config.grammar (optional GBNF), temperature, and a fixed GENERATION_SEED = 0. For temperature 0.0 (the only value the high-level surfaces use) we attach LlamaSampler::greedy() after the optional grammar. For non-zero temperatures we attach temp then dist with the seed.
8. Per-token loop. Until either the model emits an EOG / EOS token, or max_tokens is hit, or a stop sequence appears in the running output:
   • Sample, then check is_eog_token and token_eos (Qwen's chat templates use both).
   • Detokenise the new token via a UTF-8 encoding_rs decoder so multi-byte sequences split across token boundaries do not garble.
   • Push to the running String, accept the token in the sampler.
   • Test for any of config.stop_sequences in the running buffer; truncate and break if one is found.
   • Otherwise re-batch the new token alone and ctx.decode it for the next round.
9. Trim and return. Ok(generated.trim().to_string()).

High-level surfaces (also on LlmEngine)

Each is documented in its own page:

• cleanup_text — crates/llm/src/lib.rs:232
• decompose_task and decompose_task_with_feedback — :254, :267
• extract_tasks and extract_tasks_with_feedback — :294, :358
• extract_content_tags — :306

All four set temperature: 0.0 and pass at least one llama-cpp stop sequence (<|im_end|> and <|im_end_of_text|>). All four call render_chat_prompt to apply the model's tokenizer-bundled chat template, with a ChatML fallback.

render_chat_prompt (crates/llm/src/lib.rs:462)

Tries model.chat_template(None). If the GGUF carries a tokenizer-defined template, that is used. Otherwise we log a tracing::warn! and fall back to a built-in LlamaChatTemplate::new("chatml"). ChatML is what the Qwen3.5 / 3.6 family expects, so the fallback is safe in practice; the warn is there for the day someone tries a model from outside the registered family.

parse_string_array (crates/llm/src/lib.rs:489)

Helper that the array-returning surfaces share. Calls serde_json::from_str::<Vec<String>>(raw.trim()), then trims items, drops empties, and dedupes case-insensitively while preserving first-seen ordering. The case-insensitive dedupe matters: the LLM occasionally emits "Buy milk" and "buy milk" in the same array, and the user does not want to see both.

build_sampler (crates/llm/src/lib.rs:400)

Composes LlamaSampler::grammar(model, grammar, "root") (when config.grammar is set), then either LlamaSampler::greedy() for temperature 0.0 or LlamaSampler::temp(temperature) plus LlamaSampler::dist(GENERATION_SEED) for non-zero temperatures. Single-element chains are returned as-is; multi-element chains use LlamaSampler::chain_simple.

Internal constants (crates/llm/src/lib.rs:23-26)

• DEFAULT_CONTEXT_TOKENS = 4096 — minimum context window we ever allocate.
• MAX_CONTEXT_TOKENS = 8192 — hard cap. Both the preflight error and the realised n_ctx are bounded here.
• CONTEXT_RESERVE_TOKENS = 64 — extra headroom subtracted from the prompt budget so a tight fit never wedges the sampler.
• GENERATION_SEED = 0 — fixed sampling seed. Combined with temperature 0.0, makes greedy decoding fully deterministic for the high-level surfaces.

Data flow

caller prompt + GenerationConfig
  → str_to_token (AddBos::Never)
  → preflight_context_window  (or PromptTooLong error)
  → tuning::inference_thread_count (with gpu_offloaded)
  → LlamaContextParams (n_ctx, n_batch, n_ubatch, n_threads)
  → LlamaModel::new_context
  → LlamaBatch (prefill, last token logits)
  → ctx.decode
  → loop:
       LlamaSampler::sample (greedy or temp+dist, optional grammar)
       → is_eog_token / token_eos check
       → token_to_piece (UTF-8 incremental decoder)
       → stop-sequence check
       → batch.clear + add(next, cursor) + ctx.decode
  → trim + return String

The high-level surfaces wrap this with: a chat-template render in front, and (for array surfaces) parse_string_array plus a typed JSON deserialise behind.

Prompts and grammars

This file does not hold prompts or grammars itself. See llm-prompts-and-grammars.md for the catalogue. The engine consumes them by reference in each surface.

Watch-outs

• Mutex-protected single model. LlmEngine allows only one model loaded at a time. Two concurrent generate calls serialise on the underlying llama context (each call builds its own new_context from the shared LlamaModel, so the model weights are shared but the KV cache is per-call). The Tauri layer wraps each high-level call in tokio::task::spawn_blocking because generate is sync and blocks the executor for hundreds of milliseconds at minimum.
• MAX_CONTEXT_TOKENS = 8192 is a process-wide cap regardless of which tier is loaded. The 27B tier's native context is much larger; we are deliberately leaving headroom on the table to keep the preflight predictable. Surfaced in the slice README's debt section.
• u32::MAX GPU offload. The engine has no concept of partial offload. On a low-VRAM machine that cannot fit all layers, llama.cpp will emit warnings and fall back to mixed CPU/GPU automatically, but our gpu_offloaded boolean tells inference_thread_count we are fully GPU-resident. When this matters (battery, throttling), the consumer is lumotia_core::tuning — it picks a bigger thread count when it thinks the CPU is idle. Trivial-true today; tracked as observability gap (commit 052265b).
• GBNF parser quirks. llama-cpp-2's GBNF is strict about whitespace. Each grammar in llm-prompts-and-grammars.md carries an explicit ws rule and r#""# raw strings — refactors that try to "tidy" the grammar literal by stripping the trailing newline have, in the past, broken LlamaSampler::grammar with cryptic parse errors.
• Stop sequences are post-detokenisation substring matches. They run on the running String, not on token ids. A multi-byte stop string that splits across a token boundary still matches because the UTF-8 decoder buffers partial bytes. A stop string that contains characters the chat template re-emits as part of normal output (e.g. a literal <|) will trigger early termination — only use the EOG sentinels we already use.
• Chat template fallback to ChatML. If model.chat_template(None) errors, we warn-log and use LlamaChatTemplate::new("chatml"). The warn-log fires once per generate call, not once per session — keep an eye on log volume if a non-Qwen model is ever loaded.
• Backend reuse across loads. LlmState.backend is intentionally not dropped on unload. Re-init of LlamaBackend after a previous init is unsupported by llama-cpp-2; we keep one alive for the lifetime of the process.

See also

• LLM cleanup_text
• LLM decompose_task
• LLM extract_tasks
• LLM extract_content_tags
• Prompts and grammars catalogue
• Model manager
• Cargo features
• Tests
• Slice README