Jake
26c7307607
Phase 9 of the rebrand cascade. Sweep covers everything the Phase 8
frontend pass deliberately skipped: docs/, root markdown, scripts,
Cargo.toml descriptions, code comments that survived earlier
word-boundary sed, plus a handful of identifiers caught on the final
verify pass.
transcription-app changes:
- README.md, HANDOVER.md, KNOWN-ISSUES.md, run.sh — magnotia/Magnotia
-> lumotia/Lumotia.
- docs/ — sweep across all subdirs except docs/handovers/ (preserved
as immutable audit trail). Includes architecture-map references
to magnotia_core::*, magnotia_storage::*, etc. now pointing at
lumotia_*; dev-setup.md tracing output examples (lumotia_startup
target); brief/ + superpowers/ + issues/ + whisper-ecosystem/ +
audit/.
- Cargo.toml descriptions on 9 crates (core, audio, cloud-providers,
hotkey, llm, mcp, plus referenced others).
- crates/core/src/{error,hardware,recommendation,paths}.rs +
crates/audio/src/wav.rs + crates/llm/src/model_manager.rs +
crates/cloud-providers/src/keystore.rs + crates/mcp/src/lib.rs —
doc comments and a model-manager user-agent string.
- Caught on final pass: BroadcastChannel("magnotia_task_sync") -> ...
("lumotia_task_sync"); magnotia_locale i18n localStorage key
renamed + migration shim added; CSS keyframe names
magnotiaPulse / magnotiaBar / magnotiaFade renamed in the design-
system kit; magnotia_viewer_item / magnotia_viewer_mode handoff
keys renamed in HistoryPage + viewer/+page.svelte; src/assets/
wordmark.svg text.
- src-tauri/src/lib.rs comment cleanup ("magnotia era" was sed'd
to "lumotia era" earlier — restored).
Preserved (intentional):
- crates/core/src/paths.rs — keeps "magnotia" / "Magnotia" / ".magnotia"
legacy detection strings in legacy_and_target_paths() so the
migration shim can still find user data from the magnotia era.
- src/lib/stores/{page,focusTimer}.svelte.ts + src/lib/i18n/index.ts
— migration call sites reference the legacy magnotia keys
deliberately.
- docs/handovers/ — historical audit trail.
cargo build --workspace passes. npm run check: 0 errors / 0 warnings
(3958 files). cargo test --workspace: 339 pass / 0 fail.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
110 lines
6.0 KiB
Markdown
110 lines
6.0 KiB
Markdown
---
|
|
name: LLM tests
|
|
type: architecture-map-page
|
|
slice: 04-llm-formatting-mcp
|
|
last_verified: 2026/05/09
|
|
---
|
|
|
|
# LLM tests
|
|
|
|
> **Where you are:** [Architecture map](../README.md) → [LLM, Formatting, MCP](README.md) → Tests
|
|
|
|
**Plain English summary.** Two integration smoke tests live under `crates/llm/tests/`. Both gate on the `MAGNOTIA_LLM_TEST_MODEL` environment variable and skip silently when it is not set. They exist to verify that real model loading and inference works end-to-end, but never run in default `cargo test` runs because model load is heavy.
|
|
|
|
## At a glance
|
|
|
|
- Crate: `lumotia-llm`
|
|
- Paths:
|
|
- `crates/llm/tests/smoke.rs` — 62 LOC
|
|
- `crates/llm/tests/content_tags_smoke.rs` — 48 LOC
|
|
- Public surface: none — they are tests.
|
|
- External deps that matter: `lumotia_llm::LlmEngine`, `LlmModelId`, `is_valid_intent`, `GenerationConfig`. Plus `[dev-dependencies] tempfile = "3"` (used by unit tests in `model_manager.rs`, not the smoke tests).
|
|
- Tauri command that calls this: n/a — tests.
|
|
|
|
## What's in here
|
|
|
|
### `smoke.rs` — `llama_cpp_2_smoke_generates_and_wraps`
|
|
|
|
Verifies the four high-level surfaces against a real loaded model.
|
|
|
|
Path-and-line summary:
|
|
|
|
- Env-var gate at `crates/llm/tests/smoke.rs:19-25`.
|
|
- Loads the 2B tier (`LlmModelId::Qwen3_5_2B_Q4`) at the path the env var supplies, with `use_gpu: true`.
|
|
- `engine.generate("Write exactly one short greeting.", ...)` with `max_tokens: 32`, `stop_sequences: ["\n"]`, no grammar. Asserts the result is non-empty.
|
|
- `engine.cleanup_text(<filler-aware system prompt>, "um hello there like general kenobi")`. Asserts non-empty.
|
|
- `engine.extract_tasks("I need to call the plumber tomorrow and buy milk.")`. Asserts non-empty.
|
|
- `engine.decompose_task("Plan a weekend trip to the coast")`. Asserts the result has between 3 and 7 items inclusive (the GBNF guarantee).
|
|
|
|
Verified-against block at the top of the file documents the llama-cpp-2 0.1.144 surface used: `LlamaBackend`, `LlamaModel`, `LlamaContextParams`, `LlamaSampler`. Useful when bumping the dependency.
|
|
|
|
### `content_tags_smoke.rs` — `extract_content_tags_returns_valid_pair`
|
|
|
|
Verifies the Phase 9 `extract_content_tags` surface against a real loaded model.
|
|
|
|
Path-and-line summary:
|
|
|
|
- Env-var gate at `crates/llm/tests/content_tags_smoke.rs:17-23`. Same `MAGNOTIA_LLM_TEST_MODEL` as `smoke.rs`.
|
|
- Loads the 2B tier (the smoke tests deliberately use the smallest tier so they run quickly even on modest hardware).
|
|
- Realistic transcript: a multi-sentence dictation about a grant application and a meeting.
|
|
- Calls `engine.extract_content_tags(transcript)` and asserts:
|
|
- `tags.topic.len() >= 3`
|
|
- every char in `tags.topic` is `is_ascii_lowercase()`, `is_ascii_digit()`, or `'-'`
|
|
- `is_valid_intent(&tags.intent)` returns true
|
|
|
|
The character-class assertion mirrors `CONTENT_TAGS_GRAMMAR`'s `topic-char ::= [a-z0-9-]` rule: if the GBNF regresses, this test catches it on real output, not just on synthetic GBNF output.
|
|
|
|
### Run command (from both files' header comments)
|
|
|
|
```bash
|
|
MAGNOTIA_LLM_TEST_MODEL=/path/to/model.gguf cargo test -p lumotia-llm \
|
|
--test content_tags_smoke -- --nocapture
|
|
```
|
|
|
|
`--test smoke` for the older test. `--nocapture` lets the env-var-not-set message surface in CI.
|
|
|
|
### Unit tests live next to source
|
|
|
|
In addition to the integration smoke tests, every source file in `crates/llm/src/` carries a `#[cfg(test)] mod tests`:
|
|
|
|
- `crates/llm/src/lib.rs:504` — `LlmEngine` lifecycle and helper functions:
|
|
- `generate_fails_when_not_loaded` (`:508`)
|
|
- `decompose_returns_error_when_not_loaded` (`:517`)
|
|
- `default_creates_unloaded_engine` (`:525`)
|
|
- `engine_is_clone_and_shares_state` (`:531`)
|
|
- `parse_string_array_trims_and_dedupes` (`:538`)
|
|
- `first_stop_index_finds_earliest_match` (`:544`)
|
|
- `prompt_preflight_rejects_oversized_prompt_tokens` (`:551`)
|
|
- `prompt_preflight_keeps_prompts_within_budget` (`:565`)
|
|
- `crates/llm/src/model_manager.rs:402` — model path, tier recommendation, `download_impl` resume + SHA verification using a TCP fixture server.
|
|
- `crates/llm/src/prompts.rs:112` — feedback prompt builder behaviour.
|
|
|
|
Default `cargo test -p lumotia-llm` runs the unit tests (no model load) plus skips both smoke tests. CI typically runs at this scope.
|
|
|
|
## Data flow (for the smoke tests)
|
|
|
|
```
|
|
env MAGNOTIA_LLM_TEST_MODEL
|
|
→ if unset: print message, return (no failure)
|
|
→ else: PathBuf
|
|
→ LlmEngine::new()
|
|
→ engine.load_model(LlmModelId::Qwen3_5_2B_Q4, &path, use_gpu: true)
|
|
→ engine.generate / cleanup_text / extract_tasks / decompose_task / extract_content_tags
|
|
→ assertions on shape (non-empty, range, character-class, closed-set)
|
|
```
|
|
|
|
## Watch-outs
|
|
|
|
- **No assertion on output content.** The smoke tests verify the contract (non-empty, JSON-array shape, character class, closed set) but never assert on what the model actually said. A model that produces semantic garbage would still pass. The shape contract is what we intentionally pin; semantic quality is reviewed by the test running engineer.
|
|
- **Smoke tests need a 2B model file.** The 2B Q4_K_M GGUF is ~1.28 GB; downloading it once is a manual prerequisite. CI is not currently configured to fetch it.
|
|
- **Smoke tests load the 2B tier specifically.** The smoke test was written against the smallest tier so it runs in a few seconds on a modest machine. Running on the 27B tier through this path would take minutes per test and would saturate VRAM.
|
|
- **No content_tags smoke for the empty-transcript error path.** The test only exercises the happy path. The error path is covered by `extract_content_tags`'s logic and would only surface here if a regression broke the typed deserialise.
|
|
- **`MAGNOTIA_LLM_TEST_MODEL` is the only env-gate.** No second gate for "I have a Vulkan-capable GPU available". `use_gpu: true` is hard-coded; on a CPU-only machine the test will still pass but run slower and produce a llama.cpp warning.
|
|
|
|
## See also
|
|
|
|
- [LLM engine](llm-engine.md)
|
|
- [LLM cleanup_text](llm-cleanup-text.md)
|
|
- [LLM extract_content_tags](llm-extract-content-tags.md)
|
|
- [Slice README](README.md)
|