Jake
26c7307607
Phase 9 of the rebrand cascade. Sweep covers everything the Phase 8
frontend pass deliberately skipped: docs/, root markdown, scripts,
Cargo.toml descriptions, code comments that survived earlier
word-boundary sed, plus a handful of identifiers caught on the final
verify pass.
transcription-app changes:
- README.md, HANDOVER.md, KNOWN-ISSUES.md, run.sh — magnotia/Magnotia
-> lumotia/Lumotia.
- docs/ — sweep across all subdirs except docs/handovers/ (preserved
as immutable audit trail). Includes architecture-map references
to magnotia_core::*, magnotia_storage::*, etc. now pointing at
lumotia_*; dev-setup.md tracing output examples (lumotia_startup
target); brief/ + superpowers/ + issues/ + whisper-ecosystem/ +
audit/.
- Cargo.toml descriptions on 9 crates (core, audio, cloud-providers,
hotkey, llm, mcp, plus referenced others).
- crates/core/src/{error,hardware,recommendation,paths}.rs +
crates/audio/src/wav.rs + crates/llm/src/model_manager.rs +
crates/cloud-providers/src/keystore.rs + crates/mcp/src/lib.rs —
doc comments and a model-manager user-agent string.
- Caught on final pass: BroadcastChannel("magnotia_task_sync") -> ...
("lumotia_task_sync"); magnotia_locale i18n localStorage key
renamed + migration shim added; CSS keyframe names
magnotiaPulse / magnotiaBar / magnotiaFade renamed in the design-
system kit; magnotia_viewer_item / magnotia_viewer_mode handoff
keys renamed in HistoryPage + viewer/+page.svelte; src/assets/
wordmark.svg text.
- src-tauri/src/lib.rs comment cleanup ("magnotia era" was sed'd
to "lumotia era" earlier — restored).
Preserved (intentional):
- crates/core/src/paths.rs — keeps "magnotia" / "Magnotia" / ".magnotia"
legacy detection strings in legacy_and_target_paths() so the
migration shim can still find user data from the magnotia era.
- src/lib/stores/{page,focusTimer}.svelte.ts + src/lib/i18n/index.ts
— migration call sites reference the legacy magnotia keys
deliberately.
- docs/handovers/ — historical audit trail.
cargo build --workspace passes. npm run check: 0 errors / 0 warnings
(3958 files). cargo test --workspace: 339 pass / 0 fail.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
226 lines
16 KiB
Markdown
226 lines
16 KiB
Markdown
---
|
|
name: Data flow end to end
|
|
type: architecture-map-data-flow
|
|
last_verified: 2026/05/09
|
|
---
|
|
|
|
# Data flow end to end
|
|
|
|
> **Where you are:** [Architecture map](README.md) → Data flow
|
|
|
|
**Plain English summary.** This page walks one recording from beginning to end. You press the hotkey, you speak, your speech becomes text, that text lands on disk in SQLite, and later it can be searched from the UI or read by an external agent over MCP. Every step names the slice that owns it and the file where the work happens.
|
|
|
|
## The shape of a session, at a glance
|
|
|
|
```
|
|
Hotkey press
|
|
│
|
|
▼
|
|
[Backend evdev listener] ──► [Tauri runtime forwards event] ──► [Frontend opens session]
|
|
│
|
|
▼
|
|
[Microphone capture]
|
|
│
|
|
▼
|
|
[Streaming resampler 16 kHz mono]
|
|
│
|
|
▼
|
|
[VAD / chunker]
|
|
│ per chunk
|
|
▼
|
|
[Whisper or Parakeet inference]
|
|
│
|
|
partial │ final
|
|
▼ ▼
|
|
[Live Channel] [Stop session]
|
|
│
|
|
▼
|
|
[Formatting pipeline]
|
|
│
|
|
▼
|
|
┌──────────────────┴──────────────────┐
|
|
▼ ▼
|
|
[Optional LLM cleanup] [Optional content tags]
|
|
│ │
|
|
└──────────────────┬──────────────────┘
|
|
▼
|
|
[Save transcript]
|
|
│
|
|
▼
|
|
[SQLite + FTS5 trigger]
|
|
│
|
|
▼
|
|
[Paste to focused window]
|
|
|
|
(later)
|
|
│
|
|
▼
|
|
[History page search] ◄─── [search_transcripts] ───► [MCP search_transcripts]
|
|
```
|
|
|
|
## Phase 1: hotkey press
|
|
|
|
**Owner:** slice 05 (`lumotia-hotkey`, Linux only) → slice 02 (`commands/hotkey.rs`).
|
|
|
|
The Linux evdev listener (`crates/hotkey/src/linux.rs`) watches `/dev/input/event*` after a one-time access check. The user-configured combo (default `Ctrl+Shift+R`, parsed from a Tauri-style string by `HotkeyCombo::from_tauri_str`) is checked against modifier state and key events. On match the listener emits `HotkeyEvent::Pressed` over an `mpsc::Sender` it was given at startup. On release it emits `HotkeyEvent::Released`. Hotplug is supported. macOS and Windows fall back to the `tauri-plugin-global-shortcut` plugin via the same command surface.
|
|
|
|
The `commands::hotkey::start_evdev_hotkey` command (`src-tauri/src/commands/hotkey.rs:67`) holds the listener inside `tauri::State` and runs a forwarder task that translates each event into a Tauri global event:
|
|
- `lumotia:hotkey-pressed` (payload `()`)
|
|
- `lumotia:hotkey-released` (payload `()`)
|
|
|
|
Slice 01's `+layout.svelte` listens for `lumotia:hotkey-pressed` and toggles recording on the active page.
|
|
|
|
## Phase 2: session opens
|
|
|
|
**Owner:** slice 01 (`DictationPage`) → slice 02 (`commands/live.rs`).
|
|
|
|
`DictationPage` opens two typed Tauri 2 channels (`tauri::ipc::Channel<LiveResultMessage>` for results, `tauri::ipc::Channel<LiveStatusMessage>` for warnings / overload / error / finished) and calls `invoke('start_live_transcription_session', ...)` with the channels passed as arguments.
|
|
|
|
`commands::live::start_live_transcription_session` (the 1 737-LOC file at `src-tauri/src/commands/live.rs`) spawns the runtime task. The runtime owns:
|
|
- a `MicrophoneCapture` from `lumotia-audio`
|
|
- a `StreamingResampler` from `lumotia-audio`
|
|
- a `WavWriter` for progressive crash-safe append
|
|
- a chunker (`RmsVadChunker` from `lumotia-transcription`)
|
|
- a `LocalEngine` reference (already loaded into `AppState` at startup or load step)
|
|
- a power assertion (`commands::power::PowerAssertion::begin`) to keep the system awake (note: no-op on Linux / Windows / macOS today, see master debt list)
|
|
|
|
## Phase 3: microphone capture
|
|
|
|
**Owner:** slice 03 (`lumotia-audio`).
|
|
|
|
`MicrophoneCapture` (`crates/audio/src/capture.rs`) opens the selected `cpal` input device. It rejects monitor sources, RMS-validates the first samples for "is this actually live audio", and forwards device hot-unplug as `CaptureRuntimeError` rather than panicking. Sample format is normalised to `f32`. Samples leave the capture thread as `AudioChunk { samples, channels, sample_rate }`.
|
|
|
|
## Phase 4: resampling
|
|
|
|
**Owner:** slice 03 (`lumotia-audio`).
|
|
|
|
`StreamingResampler` (`crates/audio/src/streaming_resample.rs`) runs `rubato` sinc resampling to 16 kHz mono. The streaming variant is used here; a separate `resample_to_16khz` (`crates/audio/src/resample.rs`) is used for file imports in phase F (file-import path, see end of this document). The 16 kHz mono PCM is what every transcription engine downstream wants. The constant `WHISPER_SAMPLE_RATE` lives in `crates/core/src/constants.rs` (slice 05) and the `pcm-processor.js` AudioWorklet in `static/` hard-codes the same number — they must move together.
|
|
|
|
## Phase 5: voice-activity detection and chunking
|
|
|
|
**Owner:** slice 03 (`lumotia-transcription`).
|
|
|
|
The Silero-based VAD module in `lumotia-audio` (`crates/audio/src/vad.rs`) is a stub today (the candidate `ort` crate revision conflicts with Parakeet's `transcribe-rs`). Live capture therefore uses an energy-based fallback, `RmsVadChunker`, in `lumotia-transcription` (`crates/transcription/src/streaming/rms_vad.rs`). It groups samples into chunks at silence boundaries with hysteresis, returning `VadChunk { samples, start_sec, end_sec }` to the runtime. WAV append happens in parallel via `WavWriter::append_chunk`.
|
|
|
|
> The `LocalAgreement` chunker and `trim_buffer_to_commit_point` (`crates/transcription/src/streaming/`) are unit-tested but not yet wired into `commands/live.rs`. Future work.
|
|
|
|
## Phase 6: inference
|
|
|
|
**Owner:** slice 03 (`lumotia-transcription`) using slice 05 (`lumotia-core`) for thread tuning.
|
|
|
|
For each completed chunk the runtime calls `LocalEngine::transcribe_sync(samples, options)`. `LocalEngine` is mutex-guarded and holds whichever backend was loaded:
|
|
- **Whisper.** `WhisperRsBackend::transcribe` (`crates/transcription/src/whisper_rs_backend.rs`) builds a fresh `WhisperState` per call, applies `set_initial_prompt` if provided, sets `n_threads` from `lumotia_core::tuning::inference_thread_count(Workload::Whisper, ...)`, runs Vulkan offload if `vulkan_loader_available()` and the binary was built with `whisper-vulkan`.
|
|
- **Parakeet.** `SpeechModelAdapter` wraps `transcribe_rs` ONNX with quantisation hard-coded to `Int8`. Same thread-count helper.
|
|
|
|
The result is `TimedTranscript { segments, language, inference_ms }`. Each `Segment` carries `{ start_time, end_time, text }`. `Segment` is defined in `crates/core/src/types.rs` (slice 05) and is the lingua franca that crosses all three transcription, formatting, and storage seams.
|
|
|
|
## Phase 7: live partial results back to UI
|
|
|
|
**Owner:** slice 02 (`commands/live.rs`) → slice 01 (`DictationPage`).
|
|
|
|
The runtime sends a `LiveResultMessage::Chunk { chunk_id, segments, raw_text, language, duration }` over the result `Channel`. `DictationPage` appends it to the visible transcript and (if the preview overlay window is open) emits `preview-append` on the Tauri event bus. The preview window listens and renders the partial.
|
|
|
|
## Phase 8: stop session, finalise
|
|
|
|
**Owner:** slice 01 → slice 02 (`commands/live.rs`).
|
|
|
|
User releases the hotkey or clicks stop. `+layout.svelte` calls `invoke('stop_live_transcription_session')`. The runtime drains its chunker, writes the WAV trailer, releases the power assertion, and emits `LiveStatusMessage::Finished` on the status channel. The accumulated `Segment[]` is the input to the next phase.
|
|
|
|
## Phase 9: formatting pipeline
|
|
|
|
**Owner:** slice 04 (`lumotia-ai-formatting`).
|
|
|
|
`post_process_segments(segments, options)` (`crates/ai-formatting/src/pipeline.rs`) runs in this order:
|
|
1. **Anti-hallucination drop.** `is_hallucination` filter rejects known Whisper noise patterns and single-token repetition (multi-token repetition is a known gap, see master debt list).
|
|
2. **Filler removal.** Configurable list of "um", "uh", "you know", etc.
|
|
3. **British English conversion.** Regex set `to_british_english`. `-ize` verbs convert; `-izing` forms slip through (known gap).
|
|
4. **Repetition collapse.** Single-token repetition collapse, separate from the anti-hallucination pass.
|
|
5. **Smart paragraph breaks.** A new paragraph starts when the gap between segments exceeds `lumotia_core::constants::SMART_PARAGRAPH_GAP_SECS`.
|
|
|
|
Output: a `String` of formatted plain text plus the structured `Segment[]`.
|
|
|
|
## Phase 10: optional LLM cleanup
|
|
|
|
**Owner:** slice 04 (`lumotia-ai-formatting::llm_client` → `lumotia-llm`).
|
|
|
|
If the user has a model loaded and "LLM cleanup" enabled, `commands::llm::cleanup_transcript_text_cmd` (`src-tauri/src/commands/llm.rs:363`) is called from the frontend. It composes:
|
|
- `CLEANUP_PROMPT` (`crates/ai-formatting/src/llm_client.rs:26`, prompt-injection-hardened)
|
|
- a per-profile dictionary suffix (read from `lumotia_storage::list_profile_terms` and `get_profile`)
|
|
- an optional style preset (`Email`, `Notes`, `Code`)
|
|
|
|
`LlmEngine::cleanup_text` runs llama-cpp-2 with `Workload::Llm` thread tuning. Output is freeform (no GBNF). The cleaned string replaces the formatted text.
|
|
|
|
## Phase 11: optional content tags
|
|
|
|
**Owner:** slice 04 (`lumotia-llm`).
|
|
|
|
Phase 9 of the brand rollout added `extract_content_tags`. The frontend can call `extract_content_tags_cmd` (`src-tauri/src/commands/llm.rs:408`). `LlmEngine::extract_content_tags` runs the `CONTENT_TAGS_SYSTEM` prompt under the `CONTENT_TAGS_GRAMMAR` GBNF and returns `ContentTags { topic, intent }` where `intent` is constrained to `INTENT_CLOSED_SET` (six values, also enforced redundantly in Rust). The result writes to `transcripts.llm_tags` (added in migration v14).
|
|
|
|
## Phase 12: optional task extraction
|
|
|
|
**Owner:** slice 04 (`lumotia-llm`) → slice 05 (`lumotia-storage`).
|
|
|
|
If the user runs "extract tasks" on the transcript, `commands::tasks::extract_tasks_from_transcript_cmd` (`src-tauri/src/commands/tasks.rs:346`) calls `LlmEngine::extract_tasks_with_feedback` under the `OPTIONAL_TASK_ARRAY_GRAMMAR` GBNF. Returned tasks are inserted via `lumotia_storage::insert_task`.
|
|
|
|
## Phase 13: persist
|
|
|
|
**Owner:** slice 02 (`commands/transcripts.rs`) → slice 05 (`lumotia-storage`).
|
|
|
|
Frontend calls `invoke('add_transcript', ...)`. `commands::transcripts::add_transcript` builds an `InsertTranscriptParams` and calls `lumotia_storage::insert_transcript` (`crates/storage/src/database.rs`). The row lands in the `transcripts` table. The `transcripts_ai` AFTER INSERT trigger (`crates/storage/src/migrations.rs`, migration v9 rebuild) updates the `transcripts_fts` external-content FTS5 table for `text` and `title`. UUIDs come from `Uuid::new_v4()` (the storage `Cargo.toml` comment still claims v7, see master debt list).
|
|
|
|
## Phase 14: paste to focused window
|
|
|
|
**Owner:** slice 02 (`commands/paste.rs`).
|
|
|
|
In parallel with persist, the frontend calls `paste_text` or `paste_text_replacing`. The 26 kB `commands/paste.rs` handles cross-platform paste, including target-window detection and pre-existing-selection replacement on Linux (Wayland handling, X11 fallback) and platform-specific clipboard plumbing.
|
|
|
|
## Phase 15: search later
|
|
|
|
**Owner:** slice 01 (`HistoryPage`) → slice 02 (`commands/transcripts.rs`) → slice 05 (`lumotia-storage`).
|
|
|
|
In `HistoryPage` the user types a query. The page calls `invoke('search_transcripts', { query })`. `commands::transcripts::search_transcripts` calls `lumotia_storage::search_transcripts` which runs a parameterised `SELECT` against `transcripts_fts MATCH ?`. Results merge with row metadata from `transcripts` and return ranked. The Porter stemmer + unicode61 + `remove_diacritics 2` tokenizer was chosen at FTS5 table creation time.
|
|
|
|
## Phase F: file import (alternative entry path)
|
|
|
|
**Owner:** slice 01 (`FilesPage`) → slice 02 (`commands/transcription.rs`) → slice 03.
|
|
|
|
If the user drops an audio file instead of dictating, the entry point is `transcribe_file` rather than the live runtime. The slice 03 path becomes:
|
|
- `decode_audio_file_limited` (symphonia) for input format support (mp3, aac, flac, pcm, vorbis, wav, ogg, isomp4)
|
|
- `resample_to_16khz` (file resampler)
|
|
- a single `LocalEngine::transcribe_sync` call over the whole buffer
|
|
- formatting + persist as in phases 9 to 14, with no live channels involved
|
|
|
|
## Phase M: MCP read by an external agent
|
|
|
|
**Owner:** slice 04 (`lumotia-mcp`) → slice 05 (`lumotia-storage::init_readonly`).
|
|
|
|
A separate `lumotia-mcp` binary (Cargo target in `crates/mcp/`) implements the Model Context Protocol revision 2024-11-05 over JSON-RPC 2.0 on stdio. At startup it opens the same SQLite database via `lumotia_storage::init_readonly` (connection mode `ro`). Four tools:
|
|
- `list_transcripts`
|
|
- `get_transcript`
|
|
- `search_transcripts`
|
|
- `list_tasks`
|
|
|
|
Each tool maps to the corresponding `lumotia_storage` function. There is no auth at the MCP layer; stdio is the trust boundary. Any future HTTP / SSE transport needs an auth design before it ships.
|
|
|
|
## Boundary contracts at a glance
|
|
|
|
| Boundary | Type | Defined in |
|
|
|---|---|---|
|
|
| Frontend ↔ Tauri | `invoke()` arg structs + return types (serde) | each `#[tauri::command]` signature in `src-tauri/src/commands/*.rs` |
|
|
| Frontend ↔ Tauri (events) | string event names + JSON payloads | listed in `01-frontend/frontend-tauri-bridge.md` and `02-tauri-runtime/README.md` |
|
|
| Live session | `tauri::ipc::Channel<LiveResultMessage>`, `Channel<LiveStatusMessage>` | `src-tauri/src/commands/live.rs` |
|
|
| Transcription engines | `Transcriber` trait + `Segment` | `crates/transcription/src/lib.rs`, `crates/core/src/types.rs` |
|
|
| Formatting input/output | `Segment[]` in, `String` out | `crates/ai-formatting/src/pipeline.rs` |
|
|
| LLM surfaces | typed Rust args, GBNF-validated JSON output, parsed into typed Rust | `crates/llm/src/lib.rs`, `crates/llm/src/grammars.rs` |
|
|
| Storage | `InsertTranscriptParams`, `TranscriptRow`, `TaskRow`, `ProfileRow`, etc | `crates/storage/src/database.rs` |
|
|
| MCP | JSON-RPC 2.0 messages over stdio | `crates/mcp/src/lib.rs` |
|
|
|
|
## See also
|
|
|
|
- [Master architecture map](README.md)
|
|
- [Frontend slice](01-frontend/README.md)
|
|
- [Tauri runtime slice](02-tauri-runtime/README.md)
|
|
- [Audio + Transcription slice](03-audio-transcription/README.md)
|
|
- [LLM + Formatting + MCP slice](04-llm-formatting-mcp/README.md)
|
|
- [Core + Storage + Hotkey + Build slice](05-core-storage-hotkey-build/README.md)
|