Jake
2491c7a7dd
Codex independent review found 11 blockers post-cascade. All addressed.
CRITICAL (data-loss / crash):
10. crates/core/src/paths.rs — migrate_legacy_data_dir_inner used
fs::rename which fails with EXDEV when source + target are on
different filesystems (encrypted-home, bind mounts, separate
$XDG_DATA_HOME partition). Combined with the Phase 5 QC fix that
made migration errors fatal, this would crash on first launch
for any user whose data dir spans filesystems. Added
rename_or_copy_tree() that falls back to copy_dir_recursive +
remove_dir_all on CrossesDevices / errno 18 (EXDEV). Symlinks
preserved verbatim. Same fallback applied to magnotia.db ->
lumotia.db inside the dir.
11. Added 4 unit tests: copy_dir_recursive preserves nested
structure, rename_or_copy_tree same-filesystem happy path,
is_cross_device classifies CrossesDevices kind + raw errno 18.
Doc residuals (blockers 1-9):
1. crates/cloud-providers/Cargo.toml — "Wyrdnote pending rebrand"
description.
2. crates/cloud-providers/src/provider.rs — module docs + test
fixture Wyrdnote refs.
3. crates/transcription/src/orchestrator.rs — module docs + test
fixture Wyrdnote refs.
4. docs/roadmap/2026-05-10-pkm-phase-tooling-shortlist.md — phase
name + outputs/wyrdnote path refs.
5. docs/architecture-map/04-llm-formatting-mcp/llm-tests.md —
MAGNOTIA_LLM_TEST_MODEL env var.
6. .../cloud-providers-stubs.md — MAGNOTIA_API_KEY_*.
7. docs/architecture-map/03-audio-transcription/tests-and-fixtures.md
— MAGNOTIA_WHISPER_*.
8. docs/gpu-tuning/plan.md — MAGNOTIA_BENCH_RUN.
9. docs/superpowers/specs/2026-05-09-battery-gpu-aware-thread-tuning-
design.md + corresponding plan — MAGNOTIA_POWER_STATE_OVERRIDE,
MAGNOTIA_INFERENCE_THREADS.
cargo test --workspace: 343 pass / 0 fail (up from 339; +4 EXDEV
fallback tests).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
110 lines
6.0 KiB
Markdown
110 lines
6.0 KiB
Markdown
---
|
|
name: LLM tests
|
|
type: architecture-map-page
|
|
slice: 04-llm-formatting-mcp
|
|
last_verified: 2026/05/09
|
|
---
|
|
|
|
# LLM tests
|
|
|
|
> **Where you are:** [Architecture map](../README.md) → [LLM, Formatting, MCP](README.md) → Tests
|
|
|
|
**Plain English summary.** Two integration smoke tests live under `crates/llm/tests/`. Both gate on the `LUMOTIA_LLM_TEST_MODEL` environment variable and skip silently when it is not set. They exist to verify that real model loading and inference works end-to-end, but never run in default `cargo test` runs because model load is heavy.
|
|
|
|
## At a glance
|
|
|
|
- Crate: `lumotia-llm`
|
|
- Paths:
|
|
- `crates/llm/tests/smoke.rs` — 62 LOC
|
|
- `crates/llm/tests/content_tags_smoke.rs` — 48 LOC
|
|
- Public surface: none — they are tests.
|
|
- External deps that matter: `lumotia_llm::LlmEngine`, `LlmModelId`, `is_valid_intent`, `GenerationConfig`. Plus `[dev-dependencies] tempfile = "3"` (used by unit tests in `model_manager.rs`, not the smoke tests).
|
|
- Tauri command that calls this: n/a — tests.
|
|
|
|
## What's in here
|
|
|
|
### `smoke.rs` — `llama_cpp_2_smoke_generates_and_wraps`
|
|
|
|
Verifies the four high-level surfaces against a real loaded model.
|
|
|
|
Path-and-line summary:
|
|
|
|
- Env-var gate at `crates/llm/tests/smoke.rs:19-25`.
|
|
- Loads the 2B tier (`LlmModelId::Qwen3_5_2B_Q4`) at the path the env var supplies, with `use_gpu: true`.
|
|
- `engine.generate("Write exactly one short greeting.", ...)` with `max_tokens: 32`, `stop_sequences: ["\n"]`, no grammar. Asserts the result is non-empty.
|
|
- `engine.cleanup_text(<filler-aware system prompt>, "um hello there like general kenobi")`. Asserts non-empty.
|
|
- `engine.extract_tasks("I need to call the plumber tomorrow and buy milk.")`. Asserts non-empty.
|
|
- `engine.decompose_task("Plan a weekend trip to the coast")`. Asserts the result has between 3 and 7 items inclusive (the GBNF guarantee).
|
|
|
|
Verified-against block at the top of the file documents the llama-cpp-2 0.1.144 surface used: `LlamaBackend`, `LlamaModel`, `LlamaContextParams`, `LlamaSampler`. Useful when bumping the dependency.
|
|
|
|
### `content_tags_smoke.rs` — `extract_content_tags_returns_valid_pair`
|
|
|
|
Verifies the Phase 9 `extract_content_tags` surface against a real loaded model.
|
|
|
|
Path-and-line summary:
|
|
|
|
- Env-var gate at `crates/llm/tests/content_tags_smoke.rs:17-23`. Same `LUMOTIA_LLM_TEST_MODEL` as `smoke.rs`.
|
|
- Loads the 2B tier (the smoke tests deliberately use the smallest tier so they run quickly even on modest hardware).
|
|
- Realistic transcript: a multi-sentence dictation about a grant application and a meeting.
|
|
- Calls `engine.extract_content_tags(transcript)` and asserts:
|
|
- `tags.topic.len() >= 3`
|
|
- every char in `tags.topic` is `is_ascii_lowercase()`, `is_ascii_digit()`, or `'-'`
|
|
- `is_valid_intent(&tags.intent)` returns true
|
|
|
|
The character-class assertion mirrors `CONTENT_TAGS_GRAMMAR`'s `topic-char ::= [a-z0-9-]` rule: if the GBNF regresses, this test catches it on real output, not just on synthetic GBNF output.
|
|
|
|
### Run command (from both files' header comments)
|
|
|
|
```bash
|
|
LUMOTIA_LLM_TEST_MODEL=/path/to/model.gguf cargo test -p lumotia-llm \
|
|
--test content_tags_smoke -- --nocapture
|
|
```
|
|
|
|
`--test smoke` for the older test. `--nocapture` lets the env-var-not-set message surface in CI.
|
|
|
|
### Unit tests live next to source
|
|
|
|
In addition to the integration smoke tests, every source file in `crates/llm/src/` carries a `#[cfg(test)] mod tests`:
|
|
|
|
- `crates/llm/src/lib.rs:504` — `LlmEngine` lifecycle and helper functions:
|
|
- `generate_fails_when_not_loaded` (`:508`)
|
|
- `decompose_returns_error_when_not_loaded` (`:517`)
|
|
- `default_creates_unloaded_engine` (`:525`)
|
|
- `engine_is_clone_and_shares_state` (`:531`)
|
|
- `parse_string_array_trims_and_dedupes` (`:538`)
|
|
- `first_stop_index_finds_earliest_match` (`:544`)
|
|
- `prompt_preflight_rejects_oversized_prompt_tokens` (`:551`)
|
|
- `prompt_preflight_keeps_prompts_within_budget` (`:565`)
|
|
- `crates/llm/src/model_manager.rs:402` — model path, tier recommendation, `download_impl` resume + SHA verification using a TCP fixture server.
|
|
- `crates/llm/src/prompts.rs:112` — feedback prompt builder behaviour.
|
|
|
|
Default `cargo test -p lumotia-llm` runs the unit tests (no model load) plus skips both smoke tests. CI typically runs at this scope.
|
|
|
|
## Data flow (for the smoke tests)
|
|
|
|
```
|
|
env LUMOTIA_LLM_TEST_MODEL
|
|
→ if unset: print message, return (no failure)
|
|
→ else: PathBuf
|
|
→ LlmEngine::new()
|
|
→ engine.load_model(LlmModelId::Qwen3_5_2B_Q4, &path, use_gpu: true)
|
|
→ engine.generate / cleanup_text / extract_tasks / decompose_task / extract_content_tags
|
|
→ assertions on shape (non-empty, range, character-class, closed-set)
|
|
```
|
|
|
|
## Watch-outs
|
|
|
|
- **No assertion on output content.** The smoke tests verify the contract (non-empty, JSON-array shape, character class, closed set) but never assert on what the model actually said. A model that produces semantic garbage would still pass. The shape contract is what we intentionally pin; semantic quality is reviewed by the test running engineer.
|
|
- **Smoke tests need a 2B model file.** The 2B Q4_K_M GGUF is ~1.28 GB; downloading it once is a manual prerequisite. CI is not currently configured to fetch it.
|
|
- **Smoke tests load the 2B tier specifically.** The smoke test was written against the smallest tier so it runs in a few seconds on a modest machine. Running on the 27B tier through this path would take minutes per test and would saturate VRAM.
|
|
- **No content_tags smoke for the empty-transcript error path.** The test only exercises the happy path. The error path is covered by `extract_content_tags`'s logic and would only surface here if a regression broke the typed deserialise.
|
|
- **`LUMOTIA_LLM_TEST_MODEL` is the only env-gate.** No second gate for "I have a Vulkan-capable GPU available". `use_gpu: true` is hard-coded; on a CPU-only machine the test will still pass but run slower and produce a llama.cpp warning.
|
|
|
|
## See also
|
|
|
|
- [LLM engine](llm-engine.md)
|
|
- [LLM cleanup_text](llm-cleanup-text.md)
|
|
- [LLM extract_content_tags](llm-extract-content-tags.md)
|
|
- [Slice README](README.md)
|