Lumotia/docs/architecture-map/data-flow-end-to-end.md

name, type, last_verified

name	type	last_verified
Data flow end to end	architecture-map-data-flow	2026/05/09

Data flow end to end

Where you are: Architecture map → Data flow

Plain English summary. This page walks one recording from beginning to end. You press the hotkey, you speak, your speech becomes text, that text lands on disk in SQLite, and later it can be searched from the UI or read by an external agent over MCP. Every step names the slice that owns it and the file where the work happens.

The shape of a session, at a glance

Hotkey press
   │
   ▼
[Backend evdev listener] ──► [Tauri runtime forwards event] ──► [Frontend opens session]
                                                                     │
                                                                     ▼
                                                         [Microphone capture]
                                                                     │
                                                                     ▼
                                                         [Streaming resampler 16 kHz mono]
                                                                     │
                                                                     ▼
                                                         [VAD / chunker]
                                                                     │  per chunk
                                                                     ▼
                                                         [Whisper or Parakeet inference]
                                                                     │
                                                          partial    │   final
                                                            ▼        ▼
                                                  [Live Channel]   [Stop session]
                                                                     │
                                                                     ▼
                                                         [Formatting pipeline]
                                                                     │
                                                                     ▼
                                                  ┌──────────────────┴──────────────────┐
                                                  ▼                                     ▼
                                       [Optional LLM cleanup]              [Optional content tags]
                                                  │                                     │
                                                  └──────────────────┬──────────────────┘
                                                                     ▼
                                                            [Save transcript]
                                                                     │
                                                                     ▼
                                                            [SQLite + FTS5 trigger]
                                                                     │
                                                                     ▼
                                                            [Paste to focused window]

                                                            (later)
                                                                     │
                                                                     ▼
                                       [History page search] ◄─── [search_transcripts] ───► [MCP search_transcripts]

Phase 1: hotkey press

Owner: slice 05 (lumotia-hotkey, Linux only) → slice 02 (commands/hotkey.rs).

The Linux evdev listener (crates/hotkey/src/linux.rs) watches /dev/input/event* after a one-time access check. The user-configured combo (default Ctrl+Shift+R, parsed from a Tauri-style string by HotkeyCombo::from_tauri_str) is checked against modifier state and key events. On match the listener emits HotkeyEvent::Pressed over an mpsc::Sender it was given at startup. On release it emits HotkeyEvent::Released. Hotplug is supported. macOS and Windows fall back to the tauri-plugin-global-shortcut plugin via the same command surface.

The commands::hotkey::start_evdev_hotkey command (src-tauri/src/commands/hotkey.rs:67) holds the listener inside tauri::State and runs a forwarder task that translates each event into a Tauri global event:

• lumotia:hotkey-pressed (payload ())
• lumotia:hotkey-released (payload ())

Slice 01's +layout.svelte listens for lumotia:hotkey-pressed and toggles recording on the active page.

Phase 2: session opens

Owner: slice 01 (DictationPage) → slice 02 (commands/live.rs).

DictationPage opens two typed Tauri 2 channels (tauri::ipc::Channel<LiveResultMessage> for results, tauri::ipc::Channel<LiveStatusMessage> for warnings / overload / error / finished) and calls invoke('start_live_transcription_session', ...) with the channels passed as arguments.

commands::live::start_live_transcription_session (the 1 737-LOC file at src-tauri/src/commands/live.rs) spawns the runtime task. The runtime owns:

• a MicrophoneCapture from lumotia-audio
• a StreamingResampler from lumotia-audio
• a WavWriter for progressive crash-safe append
• a chunker (RmsVadChunker from lumotia-transcription)
• a LocalEngine reference (already loaded into AppState at startup or load step)
• a power assertion (commands::power::PowerAssertion::begin) to keep the system awake (note: no-op on Linux / Windows / macOS today, see master debt list)

Phase 3: microphone capture

Owner: slice 03 (lumotia-audio).

MicrophoneCapture (crates/audio/src/capture.rs) opens the selected cpal input device. It rejects monitor sources, RMS-validates the first samples for "is this actually live audio", and forwards device hot-unplug as CaptureRuntimeError rather than panicking. Sample format is normalised to f32. Samples leave the capture thread as AudioChunk { samples, channels, sample_rate }.

Phase 4: resampling

Owner: slice 03 (lumotia-audio).

StreamingResampler (crates/audio/src/streaming_resample.rs) runs rubato sinc resampling to 16 kHz mono. The streaming variant is used here; a separate resample_to_16khz (crates/audio/src/resample.rs) is used for file imports in phase F (file-import path, see end of this document). The 16 kHz mono PCM is what every transcription engine downstream wants. The constant WHISPER_SAMPLE_RATE lives in crates/core/src/constants.rs (slice 05) and the pcm-processor.js AudioWorklet in static/ hard-codes the same number — they must move together.

Phase 5: voice-activity detection and chunking

Owner: slice 03 (lumotia-transcription).

The Silero-based VAD module in lumotia-audio (crates/audio/src/vad.rs) is a stub today (the candidate ort crate revision conflicts with Parakeet's transcribe-rs). Live capture therefore uses an energy-based fallback, RmsVadChunker, in lumotia-transcription (crates/transcription/src/streaming/rms_vad.rs). It groups samples into chunks at silence boundaries with hysteresis, returning VadChunk { samples, start_sec, end_sec } to the runtime. WAV append happens in parallel via WavWriter::append_chunk.

The LocalAgreement chunker and trim_buffer_to_commit_point (crates/transcription/src/streaming/) are unit-tested but not yet wired into commands/live.rs. Future work.

Phase 6: inference

Owner: slice 03 (lumotia-transcription) using slice 05 (lumotia-core) for thread tuning.

For each completed chunk the runtime calls LocalEngine::transcribe_sync(samples, options). LocalEngine is mutex-guarded and holds whichever backend was loaded:

• Whisper. WhisperRsBackend::transcribe (crates/transcription/src/whisper_rs_backend.rs) builds a fresh WhisperState per call, applies set_initial_prompt if provided, sets n_threads from lumotia_core::tuning::inference_thread_count(Workload::Whisper, ...), runs Vulkan offload if vulkan_loader_available() and the binary was built with whisper-vulkan.
• Parakeet. SpeechModelAdapter wraps transcribe_rs ONNX with quantisation hard-coded to Int8. Same thread-count helper.

The result is TimedTranscript { segments, language, inference_ms }. Each Segment carries { start_time, end_time, text }. Segment is defined in crates/core/src/types.rs (slice 05) and is the lingua franca that crosses all three transcription, formatting, and storage seams.

Phase 7: live partial results back to UI

Owner: slice 02 (commands/live.rs) → slice 01 (DictationPage).

The runtime sends a LiveResultMessage::Chunk { chunk_id, segments, raw_text, language, duration } over the result Channel. DictationPage appends it to the visible transcript and (if the preview overlay window is open) emits preview-append on the Tauri event bus. The preview window listens and renders the partial.

Phase 8: stop session, finalise

Owner: slice 01 → slice 02 (commands/live.rs).

User releases the hotkey or clicks stop. +layout.svelte calls invoke('stop_live_transcription_session'). The runtime drains its chunker, writes the WAV trailer, releases the power assertion, and emits LiveStatusMessage::Finished on the status channel. The accumulated Segment[] is the input to the next phase.

Phase 9: formatting pipeline

Owner: slice 04 (lumotia-ai-formatting).

post_process_segments(segments, options) (crates/ai-formatting/src/pipeline.rs) runs in this order:

1. Anti-hallucination drop. is_hallucination filter rejects known Whisper noise patterns and single-token repetition (multi-token repetition is a known gap, see master debt list).
2. Filler removal. Configurable list of "um", "uh", "you know", etc.
3. British English conversion. Regex set to_british_english. -ize verbs convert; -izing forms slip through (known gap).
4. Repetition collapse. Single-token repetition collapse, separate from the anti-hallucination pass.
5. Smart paragraph breaks. A new paragraph starts when the gap between segments exceeds lumotia_core::constants::SMART_PARAGRAPH_GAP_SECS.

Output: a String of formatted plain text plus the structured Segment[].

Phase 10: optional LLM cleanup

Owner: slice 04 (lumotia-ai-formatting::llm_client → lumotia-llm).

If the user has a model loaded and "LLM cleanup" enabled, commands::llm::cleanup_transcript_text_cmd (src-tauri/src/commands/llm.rs:363) is called from the frontend. It composes:

• CLEANUP_PROMPT (crates/ai-formatting/src/llm_client.rs:26, prompt-injection-hardened)
• a per-profile dictionary suffix (read from lumotia_storage::list_profile_terms and get_profile)
• an optional style preset (Email, Notes, Code)

LlmEngine::cleanup_text runs llama-cpp-2 with Workload::Llm thread tuning. Output is freeform (no GBNF). The cleaned string replaces the formatted text.

Phase 11: optional content tags

Owner: slice 04 (lumotia-llm).

Phase 9 of the brand rollout added extract_content_tags. The frontend can call extract_content_tags_cmd (src-tauri/src/commands/llm.rs:408). LlmEngine::extract_content_tags runs the CONTENT_TAGS_SYSTEM prompt under the CONTENT_TAGS_GRAMMAR GBNF and returns ContentTags { topic, intent } where intent is constrained to INTENT_CLOSED_SET (six values, also enforced redundantly in Rust). The result writes to transcripts.llm_tags (added in migration v14).

Phase 12: optional task extraction

Owner: slice 04 (lumotia-llm) → slice 05 (lumotia-storage).

If the user runs "extract tasks" on the transcript, commands::tasks::extract_tasks_from_transcript_cmd (src-tauri/src/commands/tasks.rs:346) calls LlmEngine::extract_tasks_with_feedback under the OPTIONAL_TASK_ARRAY_GRAMMAR GBNF. Returned tasks are inserted via lumotia_storage::insert_task.

Phase 13: persist

Owner: slice 02 (commands/transcripts.rs) → slice 05 (lumotia-storage).

Frontend calls invoke('add_transcript', ...). commands::transcripts::add_transcript builds an InsertTranscriptParams and calls lumotia_storage::insert_transcript (crates/storage/src/database.rs). The row lands in the transcripts table. The transcripts_ai AFTER INSERT trigger (crates/storage/src/migrations.rs, migration v9 rebuild) updates the transcripts_fts external-content FTS5 table for text and title. UUIDs come from Uuid::new_v4() (the storage Cargo.toml comment still claims v7, see master debt list).

Phase 14: paste to focused window

Owner: slice 02 (commands/paste.rs).

In parallel with persist, the frontend calls paste_text or paste_text_replacing. The 26 kB commands/paste.rs handles cross-platform paste, including target-window detection and pre-existing-selection replacement on Linux (Wayland handling, X11 fallback) and platform-specific clipboard plumbing.

Phase 15: search later

Owner: slice 01 (HistoryPage) → slice 02 (commands/transcripts.rs) → slice 05 (lumotia-storage).

In HistoryPage the user types a query. The page calls invoke('search_transcripts', { query }). commands::transcripts::search_transcripts calls lumotia_storage::search_transcripts which runs a parameterised SELECT against transcripts_fts MATCH ?. Results merge with row metadata from transcripts and return ranked. The Porter stemmer + unicode61 + remove_diacritics 2 tokenizer was chosen at FTS5 table creation time.

Phase F: file import (alternative entry path)

Owner: slice 01 (FilesPage) → slice 02 (commands/transcription.rs) → slice 03.

If the user drops an audio file instead of dictating, the entry point is transcribe_file rather than the live runtime. The slice 03 path becomes:

• decode_audio_file_limited (symphonia) for input format support (mp3, aac, flac, pcm, vorbis, wav, ogg, isomp4)
• resample_to_16khz (file resampler)
• a single LocalEngine::transcribe_sync call over the whole buffer
• formatting + persist as in phases 9 to 14, with no live channels involved

Phase M: MCP read by an external agent

Owner: slice 04 (lumotia-mcp) → slice 05 (lumotia-storage::init_readonly).

A separate lumotia-mcp binary (Cargo target in crates/mcp/) implements the Model Context Protocol revision 2024-11-05 over JSON-RPC 2.0 on stdio. At startup it opens the same SQLite database via lumotia_storage::init_readonly (connection mode ro). Four tools:

• list_transcripts
• get_transcript
• search_transcripts
• list_tasks

Each tool maps to the corresponding lumotia_storage function. There is no auth at the MCP layer; stdio is the trust boundary. Any future HTTP / SSE transport needs an auth design before it ships.

Boundary contracts at a glance

Boundary	Type	Defined in
Frontend ↔ Tauri	invoke() arg structs + return types (serde)	each #[tauri::command] signature in src-tauri/src/commands/*.rs
Frontend ↔ Tauri (events)	string event names + JSON payloads	listed in 01-frontend/frontend-tauri-bridge.md and 02-tauri-runtime/README.md
Live session	tauri::ipc::Channel<LiveResultMessage>, Channel<LiveStatusMessage>	src-tauri/src/commands/live.rs
Transcription engines	Transcriber trait + Segment	crates/transcription/src/lib.rs, crates/core/src/types.rs
Formatting input/output	Segment[] in, String out	crates/ai-formatting/src/pipeline.rs
LLM surfaces	typed Rust args, GBNF-validated JSON output, parsed into typed Rust	crates/llm/src/lib.rs, crates/llm/src/grammars.rs
Storage	InsertTranscriptParams, TranscriptRow, TaskRow, ProfileRow, etc	crates/storage/src/database.rs
MCP	JSON-RPC 2.0 messages over stdio	crates/mcp/src/lib.rs

See also

• Master architecture map
• Frontend slice
• Tauri runtime slice
• Audio + Transcription slice
• LLM + Formatting + MCP slice
• Core + Storage + Hotkey + Build slice