Jake
26c7307607
Phase 9 of the rebrand cascade. Sweep covers everything the Phase 8
frontend pass deliberately skipped: docs/, root markdown, scripts,
Cargo.toml descriptions, code comments that survived earlier
word-boundary sed, plus a handful of identifiers caught on the final
verify pass.
transcription-app changes:
- README.md, HANDOVER.md, KNOWN-ISSUES.md, run.sh — magnotia/Magnotia
-> lumotia/Lumotia.
- docs/ — sweep across all subdirs except docs/handovers/ (preserved
as immutable audit trail). Includes architecture-map references
to magnotia_core::*, magnotia_storage::*, etc. now pointing at
lumotia_*; dev-setup.md tracing output examples (lumotia_startup
target); brief/ + superpowers/ + issues/ + whisper-ecosystem/ +
audit/.
- Cargo.toml descriptions on 9 crates (core, audio, cloud-providers,
hotkey, llm, mcp, plus referenced others).
- crates/core/src/{error,hardware,recommendation,paths}.rs +
crates/audio/src/wav.rs + crates/llm/src/model_manager.rs +
crates/cloud-providers/src/keystore.rs + crates/mcp/src/lib.rs —
doc comments and a model-manager user-agent string.
- Caught on final pass: BroadcastChannel("magnotia_task_sync") -> ...
("lumotia_task_sync"); magnotia_locale i18n localStorage key
renamed + migration shim added; CSS keyframe names
magnotiaPulse / magnotiaBar / magnotiaFade renamed in the design-
system kit; magnotia_viewer_item / magnotia_viewer_mode handoff
keys renamed in HistoryPage + viewer/+page.svelte; src/assets/
wordmark.svg text.
- src-tauri/src/lib.rs comment cleanup ("magnotia era" was sed'd
to "lumotia era" earlier — restored).
Preserved (intentional):
- crates/core/src/paths.rs — keeps "magnotia" / "Magnotia" / ".magnotia"
legacy detection strings in legacy_and_target_paths() so the
migration shim can still find user data from the magnotia era.
- src/lib/stores/{page,focusTimer}.svelte.ts + src/lib/i18n/index.ts
— migration call sites reference the legacy magnotia keys
deliberately.
- docs/handovers/ — historical audit trail.
cargo build --workspace passes. npm run check: 0 errors / 0 warnings
(3958 files). cargo test --workspace: 339 pass / 0 fail.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
239 lines
14 KiB
Markdown
239 lines
14 KiB
Markdown
# Lumotia — Engineering Context for Cursor Agents
|
||
|
||
*Canonical project context for Workstream A (Codex) and Workstream B (Opus). If you are a cloud-based AI agent working on Lumotia via Cursor, read this **before** reading `brief.md` — it tells you what is already shipped, what is intentionally deferred, and the ideology rules you cannot override.*
|
||
|
||
---
|
||
|
||
## What Lumotia is
|
||
|
||
Lumotia is a local-first, cognitive-load-aware dictation + task-capture desktop app. Stack: Tauri 2 + Rust workspace + Svelte 5 frontend. Current primary target is Linux (KDE Plasma Wayland); macOS and Windows are in scope. All transcription and LLM inference run locally — no cloud call is ever made implicitly.
|
||
|
||
Key paths:
|
||
|
||
- `crates/` — Rust workspace (core, audio, transcription, llm, ai-formatting, storage, hotkey, cloud-providers, mcp)
|
||
- `src-tauri/` — Tauri application (binary + commands)
|
||
- `src/` — SvelteKit frontend (routes, lib/pages, lib/components, lib/stores)
|
||
- `docs/whisper-ecosystem/brief.md` — cross-repo research informing this workstream pair
|
||
- `HANDOVER.md`, `HANDOVER-2026-04-18.md`, `HANDOVER-2026-04-17.md` — session handovers you should skim for recent decisions
|
||
|
||
---
|
||
|
||
## Non-negotiable design rules
|
||
|
||
These are load-bearing. If a task seems to require violating one, stop and flag it; do not "just do it."
|
||
|
||
### 1. LLM / agent scope is narrow
|
||
|
||
The in-app LLM does exactly two things: **post-transcription formatting cleanup** and **task decomposition / extraction**. Nothing else.
|
||
|
||
**Do not propose or build:**
|
||
|
||
- Wake-word detection or always-listening agent flows
|
||
- Conversational / chat UI for the LLM
|
||
- Multi-provider cloud fan-out (Bedrock, Vertex, Azure, Groq, etc.)
|
||
- Agent "skills" registries or tool-use hooks driven by spoken intent
|
||
|
||
**Do keep LLM touch points focused on:**
|
||
|
||
- Formatting the transcript (done by `cleanup_transcript_text_cmd`, prompt in `crates/ai-formatting/src/llm_client.rs`)
|
||
- Decomposing a task into 3–7 physical micro-steps (`decompose_and_store`)
|
||
- Extracting action items from a transcript (`extract_tasks_from_transcript_cmd`)
|
||
- Injecting domain vocabulary as Whisper `initial_prompt` and into the cleanup prompt
|
||
|
||
Cloud provider support exists (`crates/cloud-providers`) but is empty. When it grows, ceiling is an OpenAI-compatible endpoint + Anthropic. Do not add more.
|
||
|
||
### 2. No second notes surface
|
||
|
||
Lumotia is not a notes app. The transcript surface is History + YAML-frontmatter markdown export to the user's existing notes tool (Obsidian). Transcript editing (fix recognition errors in the viewer window) is allowed. Tagging, starring, searching transcripts is allowed. Anything that starts to look like "edit long-form prose" or "sync notes to another device/cloud" is out of scope — stop and reconsider.
|
||
|
||
**Do not propose or build:**
|
||
|
||
- A Tiptap / ProseMirror / rich-text editor
|
||
- A dedicated "notes" route
|
||
- Cloud note-sync
|
||
|
||
### 3. Meeting auto-capture: opt-in, single-signal, passive
|
||
|
||
If meeting detection is touched, it must be off by default, use only process-list polling as a signal, and surface a non-modal reminder — never start recording autonomously. This is already shipped ([src-tauri/src/commands/meeting.rs](../../src-tauri/src/commands/meeting.rs), [crates/core/src/process_watch.rs](../../crates/core/src/process_watch.rs)); do not add mic-activity heuristics or calendar integration.
|
||
|
||
### 4. Raw transcript is always recoverable
|
||
|
||
The user's words as Whisper heard them are sacrosanct. LLM cleanup can run by default, but the raw transcript must always be available for revert (task #17 in `brief.md`). Do not rewrite history in the paste buffer; do not discard raw segments after cleanup.
|
||
|
||
### 5. Local-first
|
||
|
||
No feature may assume cloud availability. Offline install is the baseline; cloud is optional polish.
|
||
|
||
### 6. Low cognitive load
|
||
|
||
Lumotia is deliberately ADHD-aware. A new setting must *earn* its mental real estate. A new flow must *reduce* not add steps. When in doubt, pick the option that requires fewer user decisions.
|
||
|
||
---
|
||
|
||
## Architectural decisions (do not re-litigate)
|
||
|
||
### Whisper runtime: `whisper-rs` + Vulkan
|
||
|
||
Lumotia uses `whisper-rs` 0.16 with the `vulkan` feature ([crates/transcription/Cargo.toml](../../crates/transcription/Cargo.toml)), wrapping whisper.cpp. **Do NOT swap to faster-whisper / WhisperX / any PyTorch-based fork.** Both drag a Python runtime into the Tauri binary, which is a far larger tax than any speedup justifies. Runtime-side improvements go into whisper-rs / whisper.cpp. Model-side improvements (Distil-Whisper GGUF, Parakeet-as-English-default) are already shipped.
|
||
|
||
The `brief.md` references faster-whisper as a pattern source — treat it as one. Do not adopt it as a dependency.
|
||
|
||
### LLM runtime: `llama-cpp-2` with Qwen3.5 / Qwen3.6 tiers
|
||
|
||
`crates/llm/` owns the LLM runtime. Four tiers (Qwen3.5 2B, Qwen3.5 4B, Qwen3.5 9B, Qwen3.6 27B) selected via hardware probe. All Q4_K_M GGUFs from unsloth's HF org. Resumable HTTP downloads with SHA-256 verification. Do not add alternative runtimes (candle, mistral.rs, etc.) without strong evidence — the current stack is working.
|
||
|
||
### GPU compatibility: Vulkan everywhere
|
||
|
||
Both whisper-rs and llama-cpp-2 link the Vulkan feature. This intentionally sidesteps CUDA version hell on Windows and works across NVIDIA / AMD / Intel / macOS (via MoltenVK). Do not introduce CUDA-specific code paths.
|
||
|
||
### Hotkey backend
|
||
|
||
- **Linux (X11 and Wayland):** evdev via `crates/hotkey/src/linux.rs`. Requires the user in the `input` group; error messaging already surfaces this.
|
||
- **macOS / Windows:** Tauri's `tauri-plugin-global-shortcut`.
|
||
- Cross-platform logic lives in `src/routes/+layout.svelte`.
|
||
|
||
### `ggml` dedup (interim)
|
||
|
||
Both `llama-cpp-sys-2` and `whisper-rs-sys` statically link their own ggml. On Linux, `src-tauri/build.rs` emits `-Wl,--allow-multiple-definition` as an interim workaround. **Do not attempt to fix this in either workstream** — a proper `system-ggml` shared-lib setup is its own workstream and out of scope here.
|
||
|
||
### Window management
|
||
|
||
Tauri 2 windows with `tauri-plugin-window-state` for position/size persistence. Preview overlay (`/preview` route) is always-on-top, `skip_taskbar`, `visible_on_all_workspaces`, and has `WindowTypeHint::Utility` set via GTK on Linux. Focus-gated: only opens when the main window is unfocused at record-start.
|
||
|
||
---
|
||
|
||
## What is already shipped on `main` (DO NOT re-implement)
|
||
|
||
The `phase3-llm-runtime` branch just merged — 16 commits, summary follows. When in doubt, grep before implementing.
|
||
|
||
| Capability | File pointers |
|
||
|---|---|
|
||
| Local LLM runtime (load / unload / cleanup / extract_tasks / decompose_task) | [crates/llm/src/lib.rs](../../crates/llm/src/lib.rs), [crates/llm/src/model_manager.rs](../../crates/llm/src/model_manager.rs) |
|
||
| LLM commands (tier recommend / download / load / unload / delete / status / cleanup / extract / decompose) | [src-tauri/src/commands/llm.rs](../../src-tauri/src/commands/llm.rs), [src-tauri/src/commands/tasks.rs](../../src-tauri/src/commands/tasks.rs) |
|
||
| Whisper `initial_prompt` built from caller prompt + profile prompt + profile_terms | [src-tauri/src/commands/mod.rs::build_initial_prompt](../../src-tauri/src/commands/mod.rs) |
|
||
| Distil-Whisper Small + Large v3 entries | [crates/core/src/model_registry.rs](../../crates/core/src/model_registry.rs) |
|
||
| Parakeet-as-default-for-English (scored in recommendation) | [crates/core/src/recommendation.rs](../../crates/core/src/recommendation.rs) |
|
||
| Platform paste matrix (wtype / xdotool / ydotool / osascript / SendKeys) with Wayland focus-race mitigation | [src-tauri/src/commands/paste.rs](../../src-tauri/src/commands/paste.rs) |
|
||
| i18n scaffolding (svelte-i18n, en/es/de, language selector) | [src/lib/i18n/](../../src/lib/i18n/), [src/lib/pages/SettingsPage.svelte](../../src/lib/pages/SettingsPage.svelte) |
|
||
| MCP stdio server (read-only tools: list/search/get transcripts, list tasks) | [crates/mcp/](../../crates/mcp/) |
|
||
| Meeting auto-capture (opt-in, process-list poll, edge-triggered toast) | [src-tauri/src/commands/meeting.rs](../../src-tauri/src/commands/meeting.rs) |
|
||
| FTS5-backed transcript search | [crates/storage/src/database.rs::search_transcripts](../../crates/storage/src/database.rs) |
|
||
| Transcription preview overlay (listening → live → cleanup → final → auto-hide) | [src/routes/preview/](../../src/routes/preview/), [src-tauri/src/commands/windows.rs::open_preview_window](../../src-tauri/src/commands/windows.rs) |
|
||
| Bulk profile-terms import | [src/lib/pages/SettingsPage.svelte::addBulkVocabTerms](../../src/lib/pages/SettingsPage.svelte) |
|
||
| Per-window size + position persistence | `tauri-plugin-window-state` registered in [src-tauri/src/lib.rs](../../src-tauri/src/lib.rs) |
|
||
|
||
This means item **#18** in `brief.md` (initial prompt / custom-vocab priming) is **already shipped** — verify and skip, don't re-implement.
|
||
|
||
Partial coverage worth extending (not re-writing):
|
||
|
||
- **Hallucination blocklist (#22)** — partial via `ai-formatting::rule_based::is_hallucination`. Extend with `avg_logprob` / `no_speech_prob` gate, don't replace the regex list.
|
||
- **Engine abstraction (#13)** — partial via `LocalEngine` wrapping both whisper and parakeet paths. Unify rather than greenfield.
|
||
- **Warm-up (#23)** — model is pre-warmed via `prewarm_default_model`. Silent-WAV inference pass is the missing piece.
|
||
- **Plain-text LLM input (#29)** — already joined as text before cleanup in `ai-formatting::pipeline`. Verify timestamps are stripped; likely a one-line confirmation.
|
||
|
||
---
|
||
|
||
## Intentionally deferred (do NOT work on these)
|
||
|
||
### Voice calibration roadmap (three tiers: mic-levels → passage → continuous)
|
||
|
||
A per-user speech-gate + initial_prompt priming flow. Discussed and scoped, but **not in either current workstream**. The hardcoded speech-gate constants in [src-tauri/src/commands/live.rs](../../src-tauri/src/commands/live.rs) lines 29–34 stay as-is for now.
|
||
|
||
### OpenWhispr audit items already decided-skipped
|
||
|
||
- **AT-SPI2 terminal detection** (Ctrl+Shift+V variant) — low ROI; most modern terminals accept Ctrl+V.
|
||
- **Multi-language auto-detect + retry** — brief item not aligned with either workstream; deferred until i18n has real multilingual users.
|
||
- **Dictation panel visibility modes** (Always / When Transcribing / Hidden) — current auto-gated behaviour covers it.
|
||
- **Mic-button visual feedback polish** — cosmetic; `VisualTimer` + `UnicodeSpinner` already exist.
|
||
|
||
### Cloud-endpoint contract test
|
||
|
||
Pinned for when `lumotia-cloud-providers` actually grows a provider. Not actionable yet.
|
||
|
||
### ggml system-lib dedup
|
||
|
||
Pinned as its own future workstream.
|
||
|
||
### Speaker diarization
|
||
|
||
Deferred. If ever built, use sherpa-onnx (already linked for Parakeet).
|
||
|
||
### Dragon-style "100-passage training"
|
||
|
||
Never. Whisper has no speaker adaptation; fine-tuning is out of Lumotia's shape.
|
||
|
||
---
|
||
|
||
## Guard-rails by workstream
|
||
|
||
### Workstream A (Codex) — Systems + streaming correctness
|
||
|
||
**Your task list:** items #1, #2, #6, #7, #8, #9, #12, #13, #19, #21, #22, #23, #24, #25, #26, #28, #29 from `brief.md`. Item #18 is already shipped — verify and skip.
|
||
|
||
**You own:**
|
||
|
||
- `crates/transcription/**`
|
||
- `crates/audio/**`
|
||
- `crates/ai-formatting/src/{pipeline,rule_based}.rs`
|
||
- `src-tauri/src/commands/{models,transcription,live,audio}.rs`
|
||
- `src-tauri/tauri.conf.json` (capabilities)
|
||
- `src-tauri/build.rs` (CPU feature detection — but do not touch the ggml link-arg)
|
||
|
||
**You do NOT touch:**
|
||
|
||
- Any `src/**` file (frontend)
|
||
- `crates/ai-formatting/src/llm_client.rs`
|
||
- `src-tauri/src/commands/{paste,clipboard,hotkey,llm}.rs` — unless the change is backend-only and flagged in your workstream-A.md as a shared-contract surface for Opus to consume
|
||
- The ggml link-arg in `src-tauri/build.rs`
|
||
|
||
**Contract surfaces** to design up front so Opus can consume them:
|
||
|
||
- `get_active_compute_device` command — returns `{ engine, backend, device_name, fallback_reason? }`
|
||
- `list_gpus` command — enumerates GPUs for the Settings dropdown
|
||
- Streaming cleanup variant — `cleanup_transcript_stream` emitting `lumotia:llm-token` events, abortable via `cancel_llm`
|
||
|
||
Document these in `docs/whisper-ecosystem/workstream-A.md` before coding so Opus can stub frontends against the agreed shape.
|
||
|
||
### Workstream B (Opus) — UX + formatting layer
|
||
|
||
**Your task list:** items #3, #4, #5, #10, #11, #14, #15, #16, #17, #20, #27, #30, #31 from `brief.md`.
|
||
|
||
**You own:**
|
||
|
||
- `src/**` (all frontend)
|
||
- `src-tauri/src/commands/{paste,clipboard,hotkey,llm}.rs` — enhance, don't replace
|
||
- `crates/ai-formatting/src/llm_client.rs` — **only the `CLEANUP_PROMPT` constant** (task #16). No other code in that file.
|
||
|
||
**You do NOT touch:**
|
||
|
||
- `crates/transcription/**`
|
||
- `crates/audio/**`
|
||
- `crates/ai-formatting/src/{pipeline,rule_based}.rs`
|
||
- `src-tauri/src/commands/{models,transcription,live,audio}.rs`
|
||
|
||
**Ideology reminders that matter most for your scope:**
|
||
|
||
- **#17 raw-revert** is where rule 4 ("raw transcript always recoverable") becomes concrete. Five-second ⌘/Ctrl+Z window, replaces LLM output with raw.
|
||
- **#16 prompt framing** — the "translator from spoken to written form, not an editor trying to improve the content" framing is load-bearing for rule 1. Do not drift from it; add a regression test that asserts the framing survives refactors.
|
||
- **#15 named prompt presets** — presets extend `profile.initial_prompt`, they do not replace it. Stay compatible with the existing per-profile vocabulary flow.
|
||
- **#27 LLM test-connection** — error classification is the point. "Ollama not installed" / "port blocked" / "model not pulled" / "scope error" — never a raw URL error to the user.
|
||
|
||
**Frontend contract** for tasks that need Codex's backend:
|
||
|
||
- #14 (GPU enum) needs `list_gpus` from Codex
|
||
- #30 (streaming LLM) needs `cleanup_transcript_stream` from Codex
|
||
- #1's chip UI (not in your list but adjacent) needs `get_active_compute_device`
|
||
|
||
If Codex hasn't shipped these yet, stub the frontend behind a feature flag with a clear TODO and ship the rest of your work — do not block.
|
||
|
||
---
|
||
|
||
## Required reading, in order
|
||
|
||
1. This document (you're in it).
|
||
2. [`docs/whisper-ecosystem/brief.md`](./brief.md) — the 31-item atomic task backlog is your source of truth.
|
||
3. [`HANDOVER.md`](../../HANDOVER.md) — latest session handover, Linux/Wayland specifics.
|
||
4. The specific Lumotia file pointers referenced under your workstream's "You own" list.
|
||
|
||
Then write your `docs/whisper-ecosystem/workstream-{A,B}.md` execution plan (sequence, dependencies, contract shapes) and commit it before writing any non-doc code.
|