Lumotia/docs/architecture-map/04-llm-formatting-mcp/llm-tests.md

---
name: LLM tests
type: architecture-map-page
slice: 04-llm-formatting-mcp
last_verified: 2026/05/09
---

# LLM tests

> **Where you are:** [Architecture map](../README.md) → [LLM, Formatting, MCP](README.md) → Tests

**Plain English summary.** Two integration smoke tests live under `crates/llm/tests/`. Both gate on the `MAGNOTIA_LLM_TEST_MODEL` environment variable and skip silently when it is not set. They exist to verify that real model loading and inference works end-to-end, but never run in default `cargo test` runs because model load is heavy.

## At a glance

- Crate: `magnotia-llm`
- Paths:
  - `crates/llm/tests/smoke.rs` — 62 LOC
  - `crates/llm/tests/content_tags_smoke.rs` — 48 LOC
- Public surface: none — they are tests.
- External deps that matter: `magnotia_llm::LlmEngine`, `LlmModelId`, `is_valid_intent`, `GenerationConfig`. Plus `[dev-dependencies] tempfile = "3"` (used by unit tests in `model_manager.rs`, not the smoke tests).
- Tauri command that calls this: n/a — tests.

## What's in here

### `smoke.rs` — `llama_cpp_2_smoke_generates_and_wraps`

Verifies the four high-level surfaces against a real loaded model.

Path-and-line summary:

- Env-var gate at `crates/llm/tests/smoke.rs:19-25`.
- Loads the 2B tier (`LlmModelId::Qwen3_5_2B_Q4`) at the path the env var supplies, with `use_gpu: true`.
- `engine.generate("Write exactly one short greeting.", ...)` with `max_tokens: 32`, `stop_sequences: ["\n"]`, no grammar. Asserts the result is non-empty.
- `engine.cleanup_text(<filler-aware system prompt>, "um hello there like general kenobi")`. Asserts non-empty.
- `engine.extract_tasks("I need to call the plumber tomorrow and buy milk.")`. Asserts non-empty.
- `engine.decompose_task("Plan a weekend trip to the coast")`. Asserts the result has between 3 and 7 items inclusive (the GBNF guarantee).

Verified-against block at the top of the file documents the llama-cpp-2 0.1.144 surface used: `LlamaBackend`, `LlamaModel`, `LlamaContextParams`, `LlamaSampler`. Useful when bumping the dependency.

### `content_tags_smoke.rs` — `extract_content_tags_returns_valid_pair`

Verifies the Phase 9 `extract_content_tags` surface against a real loaded model.

Path-and-line summary:

- Env-var gate at `crates/llm/tests/content_tags_smoke.rs:17-23`. Same `MAGNOTIA_LLM_TEST_MODEL` as `smoke.rs`.
- Loads the 2B tier (the smoke tests deliberately use the smallest tier so they run quickly even on modest hardware).
- Realistic transcript: a multi-sentence dictation about a grant application and a meeting.
- Calls `engine.extract_content_tags(transcript)` and asserts:
  - `tags.topic.len() >= 3`
  - every char in `tags.topic` is `is_ascii_lowercase()`, `is_ascii_digit()`, or `'-'`
  - `is_valid_intent(&tags.intent)` returns true

The character-class assertion mirrors `CONTENT_TAGS_GRAMMAR`'s `topic-char ::= [a-z0-9-]` rule: if the GBNF regresses, this test catches it on real output, not just on synthetic GBNF output.

### Run command (from both files' header comments)

```bash
MAGNOTIA_LLM_TEST_MODEL=/path/to/model.gguf cargo test -p magnotia-llm \
    --test content_tags_smoke -- --nocapture
```

`--test smoke` for the older test. `--nocapture` lets the env-var-not-set message surface in CI.

### Unit tests live next to source

In addition to the integration smoke tests, every source file in `crates/llm/src/` carries a `#[cfg(test)] mod tests`:

- `crates/llm/src/lib.rs:504` — `LlmEngine` lifecycle and helper functions:
  - `generate_fails_when_not_loaded` (`:508`)
  - `decompose_returns_error_when_not_loaded` (`:517`)
  - `default_creates_unloaded_engine` (`:525`)
  - `engine_is_clone_and_shares_state` (`:531`)
  - `parse_string_array_trims_and_dedupes` (`:538`)
  - `first_stop_index_finds_earliest_match` (`:544`)
  - `prompt_preflight_rejects_oversized_prompt_tokens` (`:551`)
  - `prompt_preflight_keeps_prompts_within_budget` (`:565`)
- `crates/llm/src/model_manager.rs:402` — model path, tier recommendation, `download_impl` resume + SHA verification using a TCP fixture server.
- `crates/llm/src/prompts.rs:112` — feedback prompt builder behaviour.

Default `cargo test -p magnotia-llm` runs the unit tests (no model load) plus skips both smoke tests. CI typically runs at this scope.

## Data flow (for the smoke tests)

```
env MAGNOTIA_LLM_TEST_MODEL
  → if unset: print message, return (no failure)
  → else: PathBuf
       → LlmEngine::new()
       → engine.load_model(LlmModelId::Qwen3_5_2B_Q4, &path, use_gpu: true)
       → engine.generate / cleanup_text / extract_tasks / decompose_task / extract_content_tags
       → assertions on shape (non-empty, range, character-class, closed-set)
```

## Watch-outs

- **No assertion on output content.** The smoke tests verify the contract (non-empty, JSON-array shape, character class, closed set) but never assert on what the model actually said. A model that produces semantic garbage would still pass. The shape contract is what we intentionally pin; semantic quality is reviewed by the test running engineer.
- **Smoke tests need a 2B model file.** The 2B Q4_K_M GGUF is ~1.28 GB; downloading it once is a manual prerequisite. CI is not currently configured to fetch it.
- **Smoke tests load the 2B tier specifically.** The smoke test was written against the smallest tier so it runs in a few seconds on a modest machine. Running on the 27B tier through this path would take minutes per test and would saturate VRAM.
- **No content_tags smoke for the empty-transcript error path.** The test only exercises the happy path. The error path is covered by `extract_content_tags`'s logic and would only surface here if a regression broke the typed deserialise.
- **`MAGNOTIA_LLM_TEST_MODEL` is the only env-gate.** No second gate for "I have a Vulkan-capable GPU available". `use_gpu: true` is hard-coded; on a CPU-only machine the test will still pass but run slower and produce a llama.cpp warning.

## See also

- [LLM engine](llm-engine.md)
- [LLM cleanup_text](llm-cleanup-text.md)
- [LLM extract_content_tags](llm-extract-content-tags.md)
- [Slice README](README.md)