Lumotia/docs/architecture-map/04-llm-formatting-mcp/llm-tests.md

name, type, slice, last_verified

name	type	slice	last_verified
LLM tests	architecture-map-page	04-llm-formatting-mcp	2026/05/09

LLM tests

Where you are: Architecture map → LLM, Formatting, MCP → Tests

Plain English summary. Two integration smoke tests live under crates/llm/tests/. Both gate on the MAGNOTIA_LLM_TEST_MODEL environment variable and skip silently when it is not set. They exist to verify that real model loading and inference works end-to-end, but never run in default cargo test runs because model load is heavy.

At a glance

• Crate: magnotia-llm
• Paths:
  • crates/llm/tests/smoke.rs — 62 LOC
  • crates/llm/tests/content_tags_smoke.rs — 48 LOC
• Public surface: none — they are tests.
• External deps that matter: magnotia_llm::LlmEngine, LlmModelId, is_valid_intent, GenerationConfig. Plus [dev-dependencies] tempfile = "3" (used by unit tests in model_manager.rs, not the smoke tests).
• Tauri command that calls this: n/a — tests.

What's in here

smoke.rs — llama_cpp_2_smoke_generates_and_wraps

Verifies the four high-level surfaces against a real loaded model.

Path-and-line summary:

• Env-var gate at crates/llm/tests/smoke.rs:19-25.
• Loads the 2B tier (LlmModelId::Qwen3_5_2B_Q4) at the path the env var supplies, with use_gpu: true.
• engine.generate("Write exactly one short greeting.", ...) with max_tokens: 32, stop_sequences: ["\n"], no grammar. Asserts the result is non-empty.
• engine.cleanup_text(<filler-aware system prompt>, "um hello there like general kenobi"). Asserts non-empty.
• engine.extract_tasks("I need to call the plumber tomorrow and buy milk."). Asserts non-empty.
• engine.decompose_task("Plan a weekend trip to the coast"). Asserts the result has between 3 and 7 items inclusive (the GBNF guarantee).

Verified-against block at the top of the file documents the llama-cpp-2 0.1.144 surface used: LlamaBackend, LlamaModel, LlamaContextParams, LlamaSampler. Useful when bumping the dependency.

content_tags_smoke.rs — extract_content_tags_returns_valid_pair

Verifies the Phase 9 extract_content_tags surface against a real loaded model.

Path-and-line summary:

• Env-var gate at crates/llm/tests/content_tags_smoke.rs:17-23. Same MAGNOTIA_LLM_TEST_MODEL as smoke.rs.
• Loads the 2B tier (the smoke tests deliberately use the smallest tier so they run quickly even on modest hardware).
• Realistic transcript: a multi-sentence dictation about a grant application and a meeting.
• Calls engine.extract_content_tags(transcript) and asserts:
  • tags.topic.len() >= 3
  • every char in tags.topic is is_ascii_lowercase(), is_ascii_digit(), or '-'
  • is_valid_intent(&tags.intent) returns true

The character-class assertion mirrors CONTENT_TAGS_GRAMMAR's topic-char ::= [a-z0-9-] rule: if the GBNF regresses, this test catches it on real output, not just on synthetic GBNF output.

Run command (from both files' header comments)

MAGNOTIA_LLM_TEST_MODEL=/path/to/model.gguf cargo test -p magnotia-llm \
    --test content_tags_smoke -- --nocapture

--test smoke for the older test. --nocapture lets the env-var-not-set message surface in CI.

Unit tests live next to source

In addition to the integration smoke tests, every source file in crates/llm/src/ carries a #[cfg(test)] mod tests:

• crates/llm/src/lib.rs:504 — LlmEngine lifecycle and helper functions:
  • generate_fails_when_not_loaded (:508)
  • decompose_returns_error_when_not_loaded (:517)
  • default_creates_unloaded_engine (:525)
  • engine_is_clone_and_shares_state (:531)
  • parse_string_array_trims_and_dedupes (:538)
  • first_stop_index_finds_earliest_match (:544)
  • prompt_preflight_rejects_oversized_prompt_tokens (:551)
  • prompt_preflight_keeps_prompts_within_budget (:565)
• crates/llm/src/model_manager.rs:402 — model path, tier recommendation, download_impl resume + SHA verification using a TCP fixture server.
• crates/llm/src/prompts.rs:112 — feedback prompt builder behaviour.

Default cargo test -p magnotia-llm runs the unit tests (no model load) plus skips both smoke tests. CI typically runs at this scope.

Data flow (for the smoke tests)

env MAGNOTIA_LLM_TEST_MODEL
  → if unset: print message, return (no failure)
  → else: PathBuf
       → LlmEngine::new()
       → engine.load_model(LlmModelId::Qwen3_5_2B_Q4, &path, use_gpu: true)
       → engine.generate / cleanup_text / extract_tasks / decompose_task / extract_content_tags
       → assertions on shape (non-empty, range, character-class, closed-set)

Watch-outs

• No assertion on output content. The smoke tests verify the contract (non-empty, JSON-array shape, character class, closed set) but never assert on what the model actually said. A model that produces semantic garbage would still pass. The shape contract is what we intentionally pin; semantic quality is reviewed by the test running engineer.
• Smoke tests need a 2B model file. The 2B Q4_K_M GGUF is ~1.28 GB; downloading it once is a manual prerequisite. CI is not currently configured to fetch it.
• Smoke tests load the 2B tier specifically. The smoke test was written against the smallest tier so it runs in a few seconds on a modest machine. Running on the 27B tier through this path would take minutes per test and would saturate VRAM.
• No content_tags smoke for the empty-transcript error path. The test only exercises the happy path. The error path is covered by extract_content_tags's logic and would only surface here if a regression broke the typed deserialise.
• MAGNOTIA_LLM_TEST_MODEL is the only env-gate. No second gate for "I have a Vulkan-capable GPU available". use_gpu: true is hard-coded; on a CPU-only machine the test will still pass but run slower and produce a llama.cpp warning.

See also

• LLM engine
• LLM cleanup_text
• LLM extract_content_tags
• Slice README