jake/Lumotia
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: architecture map (initial 5-slice generation, 105 pages)

Browse Source 
Five-slice navigable map of the entire codebase under
docs/architecture-map/. Each slice is a self-contained
breadcrumbed sub-tree:

  01-frontend (16)              Svelte/SvelteKit UI
  02-tauri-runtime (26)         src-tauri commands + lifecycle
  03-audio-transcription (16)   audio + transcription crates
  04-llm-formatting-mcp (19)    llm, ai-formatting, mcp, cloud
  05-core-storage-hotkey-build  core, storage, hotkey, workspace,
                          (26) CI, dev glue

Plus master README.md and data-flow-end-to-end.md tracing
audio bytes from microphone to FTS5 search to MCP read.

Generated by 5 parallel subagents on 2026/05/09 against
HEAD 3c47000. Each page has YAML frontmatter, file:line code
refs, sibling cross-links, plain-English summaries.

Aggregated debt surfaced (full lists in master README):
RB-08 macOS power assertion, schema head drift v14 vs v15,
VAD blocked on ort version conflict, streaming primitives
not wired into live.rs, no prompt versioning, MCP has no
auth, cloud-providers in-memory keystore, SettingsPage
2 484 LOC, commands/live.rs 1 737 LOC, dual theme system,
brand rename to Lumenote pending across the codebase.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jars
2026-05-09 14:04:13 +01:00
parent 3c47000ea9
commit a1f3f3f134
105 changed files with 11266 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

123
docs/architecture-map/03-audio-transcription/README.md Normal file
View File

@@ -0,0 +1,123 @@
---
name: Slice 3 — Audio + Transcription
type: architecture-map-page
slice: 03-audio-transcription
last_verified: 2026/05/09
---
# Slice 3: Audio + Transcription
> **Where you are:** [Architecture map](../README.md) → Audio + Transcription
**Plain English summary.** Two Rust workspace crates form the backbone of Magnotia's speech-to-text path. `magnotia-audio` captures microphone input via `cpal`, decodes audio files via `symphonia`, resamples to 16 kHz mono via `rubato`, and writes WAV files via `hound`. `magnotia-transcription` loads Whisper or Parakeet models, runs inference, and provides streaming primitives (VAD chunking, LocalAgreement-n commit policy, commit-bounded buffer trim) so live captures stay responsive.
## At a glance
| Crate | LOC (`src/`) | Purpose |
|---|---|---|
| `magnotia-audio` | 1,533 | Capture, VAD stub, resample, decode, WAV I/O |
| `magnotia-transcription` | 2,266 (incl. tests + build) | Engines, model manager, streaming primitives |
Public crate surface (re-exports from `lib.rs`):
```rust
// magnotia-audio
pub use capture::{AudioChunk, CaptureRuntimeError, DeviceInfo, MicrophoneCapture};
pub use concurrency::decode_and_resample;
pub use decode::{decode_audio_file, decode_audio_file_limited, probe_audio_duration_secs};
pub use resample::resample_to_16khz;
pub use streaming_resample::StreamingResampler;
pub use vad::SpeechDetector;
pub use wav::{read_wav, write_wav, WavWriter};
// magnotia-transcription
pub use concurrency::run_inference;
#[cfg(feature = "whisper")]
pub use local_engine::load_whisper;
pub use local_engine::{load_parakeet, LocalEngine, SpeechModelAdapter, TimedTranscript};
pub use model_manager::{download, is_downloaded, list_downloaded, model_dir, models_dir};
pub use streaming::{
sample_index_for_seconds, trim_buffer_to_commit_point, CommitDecision, CommitPolicy,
LocalAgreement, RmsVadChunker, Token, VadChunk, VadChunker,
};
pub use transcribe_rs::SpeechModel;
pub use transcriber::{Transcriber, TranscriberCapabilities};
```
External deps that matter:
- `cpal 0.17` — microphone capture, host enumeration, stream callbacks.
- `rubato 0.15` — sinc-interpolating resampler (file + streaming).
- `symphonia 0.5` (mp3, aac, flac, pcm, vorbis, wav, ogg, isomp4) — file decode.
- `hound 3.5` — WAV reader and crash-safe append writer.
- `whisper-rs 0.16` (optional, default) — direct Whisper inference, pipes `initial_prompt`.
- `transcribe-rs 0.3` (onnx) — Parakeet wrapper.
- `reqwest 0.12` (rustls-tls, stream) + `sha2 0.10` + `futures-util 0.3` — model downloads with Range resume + SHA verify.
- `tokio 1` — `spawn_blocking` for inference and decode.
- `tracing 0.1` — backend boundary observability.
- `voice_activity_detector` / `silero-vad-rust` — **deferred** (ort 2.0.0-rc.10 vs 2.0.0-rc.12 conflict; see `audio-vad.md`).
Cargo feature matrix (`magnotia-transcription`):
| Feature | Default | Gates |
|---|---|---|
| `whisper` | yes | `whisper-rs` dep, `whisper_rs_backend` module, `load_whisper` fn |
| `whisper-vulkan` | yes | `whisper-rs/vulkan` (Vulkan GPU offload) |
| Parakeet (`transcribe-rs`) | always on | unconditional dep, no feature flag |
`magnotia-audio` has no Cargo features.
## Map of this slice
- [`audio-capture-pipeline.md`](audio-capture-pipeline.md) — `cpal` enumeration, monitor-source detection, RMS validation, hot-unplug error forwarding.
- [`audio-resampling.md`](audio-resampling.md) — `resample_to_16khz` (file) and `StreamingResampler` (live).
- [`audio-vad.md`](audio-vad.md) — `SpeechDetector` (current stub) and the ort version conflict that blocks Silero.
- [`audio-file-decoding.md`](audio-file-decoding.md) — `symphonia` decode, `decode_and_resample` async wrapper, `probe_audio_duration_secs`.
- [`audio-wav-io.md`](audio-wav-io.md) — `WavWriter` (crash-safe append) and `read_wav` / `write_wav` round-trip helpers.
- [`audio-pcm-bridge.md`](audio-pcm-bridge.md) — `static/pcm-processor.js` AudioWorklet (frontend-side fallback path).
- [`transcription-engines-overview.md`](transcription-engines-overview.md) — `Transcriber` trait, `TranscriberCapabilities`, `LocalEngine`.
- [`transcription-whisper.md`](transcription-whisper.md) — `WhisperRsBackend`, `WhisperContext`, params, `initial_prompt`, GPU offload.
- [`transcription-parakeet.md`](transcription-parakeet.md) — `SpeechModelAdapter` + `ParakeetWordGranularity`.
- [`transcription-streaming.md`](transcription-streaming.md) — `VadChunker` trait, `RmsVadChunker`, `LocalAgreement`, `trim_buffer_to_commit_point`.
- [`transcription-model-manager.md`](transcription-model-manager.md) — download flow, SHA verify, `.magnotia-verified` manifest, Range resume.
- [`transcription-concurrency.md`](transcription-concurrency.md) — `run_inference` async wrapper around `spawn_blocking`.
- [`cargo-features.md`](cargo-features.md) — feature matrix, build commands, rationale.
- [`build-tokenizers-guard.md`](build-tokenizers-guard.md) — `build.rs` Windows-MSVC-CRT guard against `tokenizers`.
- [`tests-and-fixtures.md`](tests-and-fixtures.md) — `thread_sweep.rs`, `whisper_rs_smoke.rs`, `jfk_bench.rs`, env-var gates, in-tree download server fixture.
## How this slice connects to others
- **Slice 2 (Tauri runtime)** owns `src-tauri/src/commands/{audio,transcription,live,models}.rs`. Those wrappers call into this slice via the `pub use` exports above. The live command in particular drives `MicrophoneCapture` + `StreamingResampler` + `RmsVadChunker` + `LocalAgreement` + `WavWriter` together. This crate publishes the primitives; slice 2 publishes the orchestrator.
- **Slice 4 (LLM + AI formatting)** runs after this slice produces a `Transcript`. It consumes the segment text via the storage layer, never directly. No type traffic flows back into this slice from formatting.
- **Slice 5 (core / storage / hotkey / build)** provides the shared types this slice depends on:
- `magnotia_core::error::{MagnotiaError, Result}` — error envelope.
- `magnotia_core::types::{AudioSamples, Segment, Transcript, TranscriptionOptions, EngineName, ModelId, DownloadProgress, Megabytes}`.
- `magnotia_core::constants::{WHISPER_SAMPLE_RATE, VAD_SPEECH_THRESHOLD}`.
- `magnotia_core::hardware::vulkan_loader_available` — runtime Vulkan probe used by `WhisperRsBackend`.
- `magnotia_core::tuning::{inference_thread_count, Workload}` — power-aware thread count picker.
- `magnotia_core::paths::app_paths` — `models_dir()` / `speech_model_dir()` resolution.
- `magnotia_core::model_registry::{find_model, all_models, ModelEntry, ModelFile}` — declarative model catalogue.
- Storage (slice 5) writes the produced `Transcript` to disk; this slice does not call into it directly.
## Open questions / debt
- **Silero VAD blocked on ort version conflict.** `crates/audio/src/vad.rs` is a stub that returns `true` for every input. Both `voice_activity_detector` and `silero-vad-rust` pin `ort 2.0.0-rc.10`; `transcribe-rs` (Parakeet) requires `2.0.0-rc.12`. The `RmsVadChunker` (transcription crate) is the live-capture fallback while this is unresolved.
- **VAD lives in two crates.** `audio::vad::SpeechDetector` is a single-frame predicate; `transcription::streaming::RmsVadChunker` is a stateful chunker. They share neither code nor thresholds. When Silero lands, expect consolidation.
- **Two resamplers maintained in parallel.** `resample::resample_to_16khz` (file) and `streaming_resample::StreamingResampler` (live) duplicate `SincInterpolationParameters` config. Drift between them will show up as audible mismatch between file imports and live captures.
- **Streaming primitives not yet wired into `live.rs`.** Comments in `streaming/mod.rs`, `buffer_trim.rs`, and `commit_policy.rs` flag that the integration into `src-tauri/src/commands/live.rs` ships as follow-up commits. Slice 2 is the place to verify whether that has landed.
- **`WhisperRsBackend` builds a fresh `WhisperState` per call.** Comment notes "state can be reused, but fresh-per-call is simpler". Worth measuring once we have the live-streaming path producing many small calls per session.
- **Parakeet quantisation is hardcoded to `Int8`** (`local_engine::load_parakeet`). No pathway for selecting a different quant.
- **Hardcoded `set_n_threads` in tests** (`jfk_bench.rs` uses 6) versus the production helper (`inference_thread_count(Workload::Whisper, gpu_offloaded)`). Test results are not comparable to runtime numbers without re-running with the helper-picked thread count.
- **`build.rs` panics at link time** on Windows if `tokenizers` ever enters the dep graph. This is intentional defence (Whispering v7.11.0 incident referenced in the file) but means a transitive pull of `tokenizers` from any sibling crate is a hard build break — not a soft warning. Worth surfacing in a top-level CONTRIBUTING note.
- **Symphonia feature list is fixed.** The `mp3, aac, flac, pcm, vorbis, wav, ogg, isomp4` set covers the import paths in v0.x. AAC requires patent-licensed code paths in some build configs — flagged by the brief.
## Existing in-repo docs (do not duplicate)
- `docs/whisper-ecosystem/brief.md` — top-level Whisper-ecosystem audit (the items #6, #8, #13, #19, #21, #24, #25, #26 referenced throughout this slice).
- `docs/whisper-ecosystem/workstream-A.md` — VAD / streaming roadmap (the source of `RmsVadChunker` thresholds).
- `docs/whisper-ecosystem/workstream-B.md` — UI commit/tentative contract that drives `LocalAgreement`.
- `docs/whisper-ecosystem/magnotia-context.md` — context summary for the workstreams.
- `docs/code-review-2026-04-22.md` — audit pass that flagged decode.rs RB-09 and `read_wav` filter_map bug (both fixed in tree).
- `docs/issues/decoder-partial-audio-on-error.md` — the RB-09 ticket.
- `docs/issues/native-capture-worker-join.md`, `c1-live-session-race.md`, `run-live-session-monolith.md` — slice 2's live-session debt; they reference primitives in this slice.
- `docs/brief/technology-map.md` — top-level tech map (covers all crates, not just this slice).

120
docs/architecture-map/03-audio-transcription/audio-capture-pipeline.md Normal file
View File

@@ -0,0 +1,120 @@
---
name: Audio capture pipeline (cpal)
type: architecture-map-page
slice: 03-audio-transcription
last_verified: 2026/05/09
---
# Audio capture pipeline (cpal)
> **Where you are:** [Architecture map](../README.md) → [Audio + Transcription](README.md) → Audio capture pipeline
**Plain English summary.** The microphone-capture path enumerates host input devices, picks one (default first, then non-monitor sources, monitor sources only as a last resort), validates it produces real audio energy via a 350 ms RMS sniff, then opens a `cpal` stream that converts every supported sample format to `f32` and pushes `AudioChunk`s into a bounded `mpsc` channel. Runtime errors during the stream (mic unplug, audio server crash) are forwarded on a separate channel so the live session can show a toast.
## At a glance
- Crate: `magnotia-audio`
- Path: `crates/audio/src/capture.rs`
- LOC: 583
- External deps: `cpal 0.17`, `serde 1` (DeviceInfo wire type)
- Internal callers (best effort, slice 2 reconciles): `src-tauri/src/commands/audio.rs` (device list), `src-tauri/src/commands/live.rs` (start/stop, chunk + error channels).
Public surface:
- `pub struct AudioChunk` — `crates/audio/src/capture.rs:24` — `samples: Vec<f32>`, `sample_rate: u32`, `channels: u16`.
- `pub struct DeviceInfo` — `crates/audio/src/capture.rs:33` — Serde-derived for the Tauri IPC boundary.
- `pub struct CaptureRuntimeError` — `crates/audio/src/capture.rs:58` — non-fatal stream error reported after `start()` returned.
- `pub struct MicrophoneCapture` — `crates/audio/src/capture.rs:64`.
- `pub fn MicrophoneCapture::list_devices() -> Result<Vec<DeviceInfo>>` — `crates/audio/src/capture.rs:94`.
- `pub fn MicrophoneCapture::start_with_device(name: &str) -> Result<(Self, mpsc::Receiver<AudioChunk>)>` — `crates/audio/src/capture.rs:137`.
- `pub fn MicrophoneCapture::start() -> Result<(Self, mpsc::Receiver<AudioChunk>)>` — `crates/audio/src/capture.rs:166`.
- `pub fn MicrophoneCapture::stop(&mut self)` — `crates/audio/src/capture.rs:240`.
- `pub fn MicrophoneCapture::dropped_chunks(&self) -> u64` — `crates/audio/src/capture.rs:80`.
- `pub fn MicrophoneCapture::take_error_rx(&mut self) -> Option<mpsc::Receiver<CaptureRuntimeError>>` — `crates/audio/src/capture.rs:88`.
`MicrophoneCapture` implements `Drop` (`crates/audio/src/capture.rs:247`) so a panicked caller still pauses the cpal stream.
## What's in here
### Constants and tunables
- `AUDIO_CHANNEL_CAPACITY = 32` (`capture.rs:11`) — bounded capacity for the chunk channel.
- `DEVICE_VALIDATION_MS = 350` (`capture.rs:15`) — how long the sniffer listens before deciding a device is real.
- `SILENCE_RMS_FLOOR = 1e-5` (`capture.rs:21`) — lower bound for "produced real audio" during validation.
- `DEAD_SILENCE_FLOOR = 1e-7` (`capture.rs:475`) — even fallback (monitor) sources must clear this; pure dead-zero is rejected.
### Device selection (auto)
`start()` (`capture.rs:166`) sorts every input device into four tiers, tries each in order, and stops at the first that passes RMS validation:
1. Default + non-monitor (key 0).
2. Any other non-monitor (key 1).
3. Default but is a monitor (key 2, very rare).
4. Monitor source last resort (key 3).
The first pass enforces `require_audio = true`. If nothing clears the silence floor the loop does a second pass with `require_audio = false`, which still rejects dead silence but accepts a monitor source as a fallback.
Monitor detection (`is_monitor_name`, `capture.rs:258`) catches the standard PulseAudio / PipeWire patterns: `.monitor` suffix, `Monitor of ` prefix, anything containing `loopback`.
### Device selection (explicit)
`start_with_device()` (`capture.rs:137`) takes an exact device name (matched against `cpal::Device::description().name()`) and refuses to fall back. It still runs the same RMS validation. Returns a clear "open Settings → Audio" message if the requested device is no longer enumerated.
### ALSA description enrichment (Linux only)
`load_alsa_card_descriptions()` (`capture.rs:297`) parses `/proc/asound/cards` to map cpal's terse short name (`Microphones`) to the human-readable product string (`Blue Microphones`). Empty map on non-Linux. `extract_card_id()` (`capture.rs:278`) pulls `CARD=...` out of an ALSA device string.
### Stream construction
`open_and_validate()` (`capture.rs:351`) is the single entry point that:
1. Reads `default_input_config()` for sample rate, channel count, sample format.
2. Creates a `mpsc::sync_channel::<AudioChunk>(AUDIO_CHANNEL_CAPACITY)`.
3. Creates a `mpsc::sync_channel::<CaptureRuntimeError>(32)` for runtime errors.
4. Dispatches to `build_input_stream::<T>` for the device's sample format (F32 / I16 / U16). Anything else returns `MagnotiaError::AudioCaptureFailed`.
5. Calls `stream.play()`.
6. Sniffs samples for `DEVICE_VALIDATION_MS`, sums squared samples, derives RMS.
7. Rejects below floor (or dead silence). Otherwise re-queues the validation chunks back into the channel so downstream consumers do not lose the first 350 ms.
`build_input_stream::<T>` (`capture.rs:505`) is generic over `T: Sample + SizedSample` with `f32: FromSample<T>` so the same body handles all three sample formats. The data callback maps every sample to `f32`, packages an `AudioChunk`, and `try_send`s on the channel; failure increments the `dropped_chunks` atomic. The error callback ships a `CaptureRuntimeError` on `err_tx` (channel-full path increments `dropped_errors` and logs to stderr).
### Drop counters
Two atomics give the live session visibility:
- `dropped_chunks: Arc<AtomicU64>` — chunk channel was full. Diagnostic for downstream backpressure.
- `dropped_errors: Arc<AtomicU64>` (private) — runtime-error channel was full (caller stopped draining). Logged to stderr each time.
## Data flow
```
cpal default_host()
└─ input_devices() (enumerate)
└─ open_and_validate(device)
├─ build_input_stream::<f32|i16|u16>(...)
│ └─ data callback: T → f32 → AudioChunk → mpsc::sync_channel
│ └─ error callback: cpal::StreamError → CaptureRuntimeError → err_mpsc
└─ stream.play()
└─ 350 ms sniff → RMS → accept | reject
└─ on accept: re-queue collected chunks, return MicrophoneCapture + Receiver<AudioChunk>
```
Output: one `AudioChunk` per cpal callback period at the device's native rate. The live session is responsible for downmixing channels (if `channels > 1`) and feeding `StreamingResampler` to reach 16 kHz mono. Native rate is *not* normalised here.
## Watch-outs
- **Channel capacity is 32 chunks.** At a 1024-frame cpal buffer at 48 kHz that's roughly 700 ms. A blocked consumer for longer than that means dropped audio. `dropped_chunks()` is the visibility hook; the live-session command must surface it.
- **Default-device first works against the safest pick on Linux Pulse setups** where the default sink monitor sneaks in. The four-tier sort handles this, but only because monitor names match the patterns in `is_monitor_name`. New PipeWire schemes that don't include `.monitor` / `loopback` would slip through.
- **350 ms validation window adds a startup latency floor.** Slice 2 needs to know about this when wiring "click record".
- **`stop()` is `pause`, not `drop`.** The stream object is kept alive until `Drop`. A subsequent `start()` on the same `MicrophoneCapture` is not supported (signature returns a fresh instance).
- **Sample format dispatch is closed-set.** Anything not F32 / I16 / U16 is a hard error. cpal can in principle expose I8 / I32 / F64 on exotic devices.
- **`device_display_name` swallows errors.** `cpal::Device::description()` errors silently become `None`, then `<unnamed>` downstream. Acceptable for a UI list, surprising for debugging.
- **Re-queue uses `try_send` on a channel of capacity 32.** If the sniff produced more than 32 chunks (≈64 ms at 48 kHz 256-frame buffers — uncommon but possible), the early ones are dropped against the same `dropped_chunks` counter. Documented at `capture.rs:486`.
## See also
- [Audio resampling](audio-resampling.md) — the live session feeds `AudioChunk.samples` into `StreamingResampler` to get to 16 kHz.
- [Audio VAD](audio-vad.md) — what happens to chunks once they're at 16 kHz.
- [Audio WAV I/O](audio-wav-io.md) — `WavWriter` is the crash-safe sibling that persists capture audio to disk.
- [Audio PCM bridge](audio-pcm-bridge.md) — the static AudioWorklet that exists for browser-side fallback paths.
- Tests at `crates/audio/src/capture.rs:567` cover monitor pattern detection. The validation-window logic is exercised end-to-end through the live integration tests in slice 2.

86
docs/architecture-map/03-audio-transcription/audio-file-decoding.md Normal file
View File

@@ -0,0 +1,86 @@
---
name: Audio file decoding (symphonia)
type: architecture-map-page
slice: 03-audio-transcription
last_verified: 2026/05/09
---
# Audio file decoding (symphonia)
> **Where you are:** [Architecture map](../README.md) → [Audio + Transcription](README.md) → Audio file decoding
**Plain English summary.** When the user imports an audio file, `decode_audio_file` reads it through `symphonia`, downmixes to mono, and returns f32 PCM at the file's native rate. `decode_and_resample` chains this with `resample_to_16khz` on a blocking task so the Tauri command stays async. A short `probe_audio_duration_secs` helper reads the duration without decoding the body, used for "is this longer than the import limit?" preflight.
## At a glance
- Crate: `magnotia-audio`
- Paths: `crates/audio/src/decode.rs` (283 LOC), `crates/audio/src/concurrency.rs` (19 LOC).
- External deps: `symphonia 0.5` with features `mp3, aac, flac, pcm, vorbis, wav, ogg, isomp4`.
- Internal callers (best effort, slice 2 reconciles): the import-file Tauri command and the file-mode transcription command call `decode_and_resample`. Standalone `decode_audio_file` is exposed for tests and any path that wants the native rate.
Public surface:
- `pub fn decode_audio_file(path: &Path) -> Result<AudioSamples>` — `crates/audio/src/decode.rs:23`.
- `pub fn decode_audio_file_limited(path: &Path, max_duration_secs: Option<f64>) -> Result<AudioSamples>` — `crates/audio/src/decode.rs:27`.
- `pub fn probe_audio_duration_secs(path: &Path) -> Result<Option<f64>>` — `crates/audio/src/decode.rs:43`.
- `pub async fn decode_and_resample(path: &Path) -> Result<AudioSamples>` — `crates/audio/src/concurrency.rs:11`.
## What's in here
### `decode_audio_file` and friends
`decode_audio_file` (`decode.rs:23`) delegates to `decode_audio_file_limited(path, None)`. The `_limited` variant accepts a hard cap in seconds; once the per-track sample count crosses `(max_duration_secs * sample_rate).ceil()` it returns `MagnotiaError::AudioDecodeFailed("Audio is longer than the {:.0} minute import limit", ...)`. Used by the Tauri import command to enforce the configured maximum.
Both call into `decode_media_stream` (`decode.rs:77`), which is the actual decode loop and the seam tests inject a `MediaSource` into. Steps:
1. Probe the format (`symphonia::default::get_probe()`).
2. Pick the default track; reject if there is none or the codec sample rate is `0`.
3. Build a decoder via `symphonia::default::get_codecs()`.
4. Loop:
- `format.next_packet()` — `UnexpectedEof` is the natural EOF marker; any other error is propagated as `AudioDecodeFailed`. `ResetRequired` is treated as a discontinuity error.
- Skip packets that don't belong to the chosen track.
- Decode into a `SampleBuffer<f32>`.
- Average channels into mono if `channels > 1`; otherwise extend the output verbatim.
- Apply the duration cap if one was provided.
5. Reject empty output (`No audio data decoded`) and return `AudioSamples::new(samples, sample_rate, 1)`.
### `probe_audio_duration_secs`
`decode.rs:43`. Re-runs the probe step only, reads `track.codec_params.n_frames` and `sample_rate`, returns `Some(seconds)` if both are available. Cheaper than a full decode for "is this file too long?" gating.
### `decode_and_resample`
`concurrency.rs:11`. Wraps `decode_audio_file` + `resample_to_16khz` in `tokio::task::spawn_blocking`. Used by every async caller that needs ready-to-transcribe samples. Maps `JoinError` to `MagnotiaError::AudioDecodeFailed("Task join error: {e}")`.
## Data flow
```
Path
└─ File::open
└─ MediaSourceStream
└─ Hint(extension)
└─ probe → format + codec_params
└─ Decoder
└─ next_packet loop
├─ packet → decoded → SampleBuffer<f32>
└─ if channels > 1: mean across channels (mono mixdown)
→ samples (f32, native rate)
```
`decode_and_resample` chains the above with `resample_to_16khz` so the caller gets `AudioSamples` already at 16 kHz.
## Watch-outs
- **RB-09 regression (2026-04-22 review).** The previous loop did `Err(_) => break` and silently returned partial audio on a mid-stream I/O error. Current code distinguishes EOF, `ResetRequired`, and other errors. Test at `decode.rs:262` injects a flaky `MediaSource` to keep this honest.
- **AAC and ISO MP4 are pulled in unconditionally** by the symphonia feature set. Patent-licence implications for AAC are flagged in the brief; this is not a per-build decision today.
- **Channels are averaged, not mixed psychoacoustically.** Stereo to mono goes via mean of all channels per sample. Fine for ASR; lossy for any future task that needs spatial cues.
- **Whole-file decode loads everything into memory.** The duration cap is the only guard. Long imports exceed RAM long before they exceed disk.
- **`Hint` only carries extension.** A `.mp3` named `.bin` will fail to probe even if the bytes are valid MP3. Caller responsibility to keep extensions truthful.
- **`decode_and_resample` returns the resampled output.** If a downstream caller wanted the original rate (for a non-ASR purpose), it must call `decode_audio_file` directly.
## See also
- [Audio resampling](audio-resampling.md) — `resample_to_16khz` is the second half of `decode_and_resample`.
- [Audio WAV I/O](audio-wav-io.md) — `read_wav` is the WAV-only fast path that uses `hound` instead of symphonia.
- `magnotia_core::types::AudioSamples` (slice 5) — the value type returned.
- Tests at `decode.rs:175` (RB-09 regression among others).

96
docs/architecture-map/03-audio-transcription/audio-pcm-bridge.md Normal file
View File

@@ -0,0 +1,96 @@
---
name: PCM bridge (AudioWorklet)
type: architecture-map-page
slice: 03-audio-transcription
last_verified: 2026/05/09
---
# PCM bridge (AudioWorklet)
> **Where you are:** [Architecture map](../README.md) → [Audio + Transcription](README.md) → PCM bridge
**Plain English summary.** `static/pcm-processor.js` is a tiny browser-side AudioWorklet. It runs inside the Svelte frontend's `AudioContext`, naively downsamples the device's native rate to 16 kHz, and posts ~0.5 s batches of f32 samples back to the main thread. It only matters on a code path that goes browser-mic → frontend → Tauri command rather than the native cpal capture; today's primary path is `MicrophoneCapture` (Rust). Treat this file as a minimal fallback / web-context stub.
## At a glance
- Runtime: browser AudioWorkletProcessor (not a Rust crate).
- Path: `static/pcm-processor.js`
- LOC: 44
- External deps: none (Web Audio API).
- Internal callers (best effort, slice 1 reconciles): the Svelte frontend instantiates it with `audioContext.audioWorklet.addModule("/pcm-processor.js")` and listens for `port.message` events. The native cpal path in `magnotia-audio` is the production path; this exists for parity in dev/web contexts.
Public surface: registers the AudioWorklet processor name `pcm-processor`.
## What's in here
The whole file:
```js
class PcmProcessor extends AudioWorkletProcessor {
constructor() {
super();
this.buffer = [];
this.ratio = sampleRate / 16000;
this.needsResample = Math.abs(this.ratio - 1.0) > 0.01;
this.resamplePos = 0;
}
process(inputs) {
const input = inputs[0];
if (!input || input.length === 0) return true;
const samples = input[0]; // First channel (mono)
if (!samples) return true;
if (this.needsResample) {
// Simple downsampling to 16kHz
for (let i = 0; i < samples.length; i++) {
this.resamplePos += 1;
if (this.resamplePos >= this.ratio) {
this.buffer.push(samples[i]);
this.resamplePos -= this.ratio;
}
}
} else {
for (let i = 0; i < samples.length; i++) {
this.buffer.push(samples[i]);
}
}
if (this.buffer.length >= 8000) {
this.port.postMessage({ type: "pcm", samples: this.buffer });
this.buffer = [];
}
return true;
}
}
registerProcessor("pcm-processor", PcmProcessor);
```
`sampleRate` is the AudioWorklet global (the context's rate). The file picks the first channel and ignores the rest.
## Data flow
```
mediaStream → MediaStreamAudioSourceNode → AudioWorkletNode("pcm-processor")
└─ process(inputs)
├─ if needsResample (ratio != 1): drop-sample downsample to 16 kHz
└─ else: passthrough
└─ buffer to ~0.5 s (8000 samples)
└─ postMessage({ type: "pcm", samples: f32[] })
└─ frontend receiver (slice 1) → Tauri command (slice 2)
```
## Watch-outs
- **Drop-sample downsampling, no anti-alias filter.** Above ~8 kHz the result will alias. For ASR this is rarely a problem because Whisper expects 16 kHz inputs (so anything above 8 kHz Nyquist is already discarded), but if anyone repurposes this for non-ASR work it will show.
- **Mono only.** Reads channel 0, ignores channels 1+.
- **`Math.abs(ratio - 1) > 0.01` decides passthrough.** A 16001 Hz context (rare but legal) would skip the resample loop.
- **`port.postMessage` clones the array.** No transferables in this implementation; large session lengths magnify the cost.
- **The native cpal path is the primary path.** This worklet matters when the frontend captures audio itself (web build, dev preview without the Tauri shell). Slice 1 will know which routes mount it.
- **Build artefacts mirror the source.** Copies appear under `build/` and `.svelte-kit/output/client/` after a build. The source-of-truth file is `static/pcm-processor.js`.
## See also
- [Audio capture pipeline](audio-capture-pipeline.md) — the production path using cpal, not this worklet.
- [Audio resampling](audio-resampling.md) — the (much higher quality) sinc resampler used on the native path.
- Frontend Audio plumbing (slice 1) — the consumer of `port.postMessage`.

106
docs/architecture-map/03-audio-transcription/audio-resampling.md Normal file
View File

@@ -0,0 +1,106 @@
---
name: Audio resampling (rubato)
type: architecture-map-page
slice: 03-audio-transcription
last_verified: 2026/05/09
---
# Audio resampling (rubato)
> **Where you are:** [Architecture map](../README.md) → [Audio + Transcription](README.md) → Audio resampling
**Plain English summary.** Whisper and Parakeet want 16 kHz mono `f32`. Microphones and audio files almost never deliver that natively. Two `rubato`-backed resamplers convert: `resample_to_16khz` for one-shot file conversion, `StreamingResampler` for the live session where samples arrive in arbitrary chunks and a flush at the end drains the resampler's tail.
## At a glance
- Crate: `magnotia-audio`
- Paths: `crates/audio/src/resample.rs` (100 LOC), `crates/audio/src/streaming_resample.rs` (211 LOC).
- External deps: `rubato 0.15` (sinc interpolation).
- Internal callers (best effort, slice 2 reconciles): `concurrency::decode_and_resample` (file path used by `audio.rs` and `transcription.rs` Tauri commands), `live.rs` Tauri command (live path).
Public surface:
- `pub fn resample_to_16khz(audio: &AudioSamples) -> Result<AudioSamples>` — `crates/audio/src/resample.rs:11`.
- `pub enum StreamingResampler` — `crates/audio/src/streaming_resample.rs:37`. Variants: `Passthrough`, `Sinc { resampler, residual, ratio }`.
- `pub fn StreamingResampler::new(from_rate: u32) -> Result<Self>` — `crates/audio/src/streaming_resample.rs:52`.
- `pub fn StreamingResampler::push_samples(&mut self, mono: &[f32]) -> Result<Vec<f32>>` — `crates/audio/src/streaming_resample.rs:93`.
- `pub fn StreamingResampler::flush(&mut self) -> Result<Vec<f32>>` — `crates/audio/src/streaming_resample.rs:127`.
## What's in here
### Shared sinc parameters
Both files use the same `SincInterpolationParameters`:
```rust
sinc_len: 256,
f_cutoff: 0.95,
oversampling_factor: 128,
interpolation: SincInterpolationType::Cubic,
window: WindowFunction::Blackman,
```
`max_relative_jitter = 1.1`, `chunks_in = 1024`, channels = 1.
### `resample_to_16khz` (file)
`crates/audio/src/resample.rs:11`. Pure function over an immutable `AudioSamples`. Short-circuits the passthrough case (already 16 kHz). Iterates the input in 1024-sample chunks; pads the trailing partial chunk to `chunk_size` zeros; truncates the output to `(input_len * ratio) as usize` to discard the padding's residue.
Errors map to `MagnotiaError::AudioDecodeFailed` (one variant for ratio init failure, one for per-chunk process failure).
### `StreamingResampler` (live)
`crates/audio/src/streaming_resample.rs`. Two states:
- `Passthrough` — emitted when `from_rate == WHISPER_SAMPLE_RATE`. `push_samples` returns input verbatim; `flush` returns empty.
- `Sinc { resampler, residual, ratio }` — wraps a `SincFixedIn::<f32>`. `residual` buffers samples that don't yet form a complete `INPUT_CHUNK = 1024`. Each `push_samples` call drains as many full chunks as it can.
`flush()` is the load-bearing tail handler:
1. If `residual` is empty, return empty (no flush work pending).
2. Pad `residual` up to `INPUT_CHUNK` zeros.
3. Run `resampler.process()` once over the padded chunk.
4. Truncate the output to `(leftover_real_samples * ratio).round() as usize` so the silence introduced by padding does not leak into the saved WAV.
The `flush` correctness math is verified by the `streaming_48k_to_16k_preserves_duration` test (`streaming_resample.rs:183`).
## Data flow
File path:
```
AudioSamples (any rate, mono)
└─ resample_to_16khz()
├─ if rate == 16k → AudioSamples::mono_16khz(clone)
└─ else: rubato SincFixedIn loop, padded final chunk, truncate to expected_len
→ AudioSamples (16k, mono)
```
Live path:
```
cpal AudioChunk (native rate, possibly stereo, downmixed by caller)
└─ StreamingResampler::push_samples(&mono_f32)
├─ residual.extend(mono)
├─ while residual.len() >= 1024: resampler.process(chunk) → out
└─ return out (zero or more 16 kHz samples)
...session ends...
└─ StreamingResampler::flush()
└─ pad residual to 1024, process, trim padding-induced output
→ 16 kHz mono Vec<f32>
```
## Watch-outs
- **Two implementations, one parameter set.** Drift between `resample.rs` and `streaming_resample.rs` is currently held by convention only. Any change to `SincInterpolationParameters` must touch both files.
- **Caller is responsible for downmixing.** Both functions assume mono input. `decode.rs` averages channels for files; the live session must do the same before pushing to `StreamingResampler`.
- **`from_rate = 0` is rejected** in both functions (matches a corrupt WAV header path).
- **Truncation policy differs slightly.** `resample_to_16khz` uses `(samples.len() as f64 * ratio) as usize` (truncating cast), `StreamingResampler::flush` uses `.round() as usize`. Off by one sample at most; matters for WAV file length parity if you ever compare them.
- **No streaming for file path.** `resample_to_16khz` materialises the whole resampled vec in memory. A 90-minute meeting at 48 kHz pre-resample is ~250 MB of `f32`; the post-resample buffer is another 350 MB. Acceptable today, brittle for very long imports.
- **Passthrough still calls `to_vec()`** on the input samples (`streaming_resample.rs:95`). Cheap but not free; if push frequency is high consider returning `Cow`.
## See also
- [Audio capture pipeline](audio-capture-pipeline.md) — produces the `AudioChunk`s that feed `StreamingResampler`.
- [Audio file decoding](audio-file-decoding.md) — `decode_and_resample` chains decode + `resample_to_16khz`.
- `magnotia_core::constants::WHISPER_SAMPLE_RATE` (slice 5) — single source of truth for the target rate.

55
docs/architecture-map/03-audio-transcription/audio-vad.md Normal file
View File

@@ -0,0 +1,55 @@
---
name: Audio VAD (currently a stub)
type: architecture-map-page
slice: 03-audio-transcription
last_verified: 2026/05/09
---
# Audio VAD (currently a stub)
> **Where you are:** [Architecture map](../README.md) → [Audio + Transcription](README.md) → Audio VAD
**Plain English summary.** `magnotia-audio` exposes a `SpeechDetector` type, but right now its `is_speech()` returns `true` for every input. This is a deliberate stub: both candidate Silero VAD bindings pin an older `ort` version than `transcribe-rs` (Parakeet) requires, so adding either today would force a workspace-wide downgrade. Live capture uses the energy-based `RmsVadChunker` from the transcription crate as the working fallback until the `ort` ecosystem aligns.
## At a glance
- Crate: `magnotia-audio`
- Path: `crates/audio/src/vad.rs`
- LOC: 35
- External deps: none (pulls `VAD_SPEECH_THRESHOLD` from `magnotia_core::constants`).
- Internal callers (best effort): unused in production today. The threshold is read but never compared, because `is_speech` always returns `true`.
Public surface:
- `pub struct SpeechDetector` — `crates/audio/src/vad.rs:14`, derives `Default`.
- `pub fn SpeechDetector::new() -> Self` — `crates/audio/src/vad.rs:19`.
- `pub fn SpeechDetector::is_speech(&self, _samples: &[f32]) -> bool` — `crates/audio/src/vad.rs:26`.
- `pub fn SpeechDetector::threshold(&self) -> f64` — `crates/audio/src/vad.rs:30`.
- `pub fn SpeechDetector::reset(&mut self)` — `crates/audio/src/vad.rs:34`.
## What's in here
The whole file is the stub plus an explanatory header:
> Both `voice_activity_detector` and `silero-vad-rust` pin ort 2.0.0-rc.10 which conflicts with transcribe-rs requiring ort 2.0.0-rc.12. When the ort ecosystem aligns (likely at 2.0.0 stable), add Silero VAD here.
>
> For now, all audio is treated as speech. This matches v0.2 behaviour (no VAD) and doesn't affect core functionality.
The threshold comes from `VAD_SPEECH_THRESHOLD` (slice 5) and is currently dead weight.
## Data flow
`SpeechDetector::is_speech(samples) -> true`. There is no flow.
## Watch-outs
- **The other VAD lives in the transcription crate.** `RmsVadChunker` (see [transcription-streaming.md](transcription-streaming.md)) is the live-capture chunker that actually decides what gets transcribed. When Silero lands here, expect to consolidate the two: the chunker becomes a thin shell around a real VAD predicate.
- **`is_speech` returning `true` is a feature, not a bug.** v0.2 had no VAD at all. The stub keeps the interface stable so callers can switch to a real Silero impl without a code change.
- **Do not reach for `voice_activity_detector` 0.x or `silero-vad-rust` 6.x without a workspace-wide ort audit.** The conflict is documented, has bitten before, and will bite again until upstream `ort` reaches a stable that both crates target.
- **`reset()` is a no-op.** The trait shape exists for future Silero state.
## See also
- [Transcription streaming](transcription-streaming.md) — `RmsVadChunker`, the actual gating today.
- `magnotia_core::constants::VAD_SPEECH_THRESHOLD` (slice 5) — wired but unused.
- `Cargo.toml` of `magnotia-audio` — the commented-out `silero-vad-rust = "6"` line documents the deferred dep.

84
docs/architecture-map/03-audio-transcription/audio-wav-io.md Normal file
View File

@@ -0,0 +1,84 @@
---
name: Audio WAV I/O (hound)
type: architecture-map-page
slice: 03-audio-transcription
last_verified: 2026/05/09
---
# Audio WAV I/O (hound)
> **Where you are:** [Architecture map](../README.md) → [Audio + Transcription](README.md) → Audio WAV I/O
**Plain English summary.** Two helpers cover the simple round-trip case: `write_wav` writes f32 samples to a 16-bit PCM WAV; `read_wav` reads them back, converting either int or float WAV payloads to f32. A third type, `WavWriter`, is the crash-safe append writer used during long live captures: it flushes the WAV header every 8000 samples (≈500 ms at 16 kHz), so a process kill leaves a valid, playable file on disk up to the last flush.
## At a glance
- Crate: `magnotia-audio`
- Path: `crates/audio/src/wav.rs`
- LOC: 287
- External deps: `hound 3.5`.
- Internal callers (best effort, slice 2 reconciles): `WavWriter` is consumed by the live-session command in `src-tauri/src/commands/live.rs` (brief item #19). `read_wav` and `write_wav` are used by the file-import command and by the audio-decode tests.
Public surface:
- `pub struct WavWriter` — `crates/audio/src/wav.rs:21`.
- `pub fn WavWriter::create(path: &Path, sample_rate: u32, channels: u16) -> Result<Self>` — `crates/audio/src/wav.rs:36`.
- `pub fn WavWriter::append(&mut self, samples: &[f32]) -> Result<()>` — `crates/audio/src/wav.rs:58`.
- `pub fn WavWriter::flush(&mut self) -> Result<()>` — `crates/audio/src/wav.rs:78`.
- `pub fn WavWriter::finalize(self) -> Result<()>` — `crates/audio/src/wav.rs:90`.
- `pub fn write_wav(path: &Path, audio: &AudioSamples) -> Result<()>` — `crates/audio/src/wav.rs:99`.
- `pub fn read_wav(path: &Path) -> Result<AudioSamples>` — `crates/audio/src/wav.rs:132`.
## What's in here
### `WavWriter` (crash-safe live capture)
Wraps a `hound::WavWriter<BufWriter<File>>` plus a `samples_since_flush` counter and a `flush_every` threshold (`DEFAULT_FLUSH_EVERY_SAMPLES = 8_000`, 500 ms at 16 kHz).
- `create` constructs a 16-bit PCM `WavSpec` from the supplied `sample_rate` / `channels`. Sample rate and channel count come from `LocalEngine::capabilities()` rather than being hardcoded — this matters once non-Whisper engines (Parakeet, future Moonshine) declare different rates.
- `append` clamps each f32 to `[-1.0, 1.0]`, scales to `i16`, writes the sample, and flushes the header every `flush_every` samples.
- `flush` calls `hound::WavWriter::flush` (which rewrites the RIFF and data chunk sizes in the header) and resets the counter.
- `finalize` writes the terminal header and consumes the writer. A dropped-without-finalize writer leaves a valid file up to the last flush; callers that care about the unflushed tail should always call finalize.
### `write_wav` and `read_wav`
`write_wav` is the simple one-shot writer over `AudioSamples`. Same clamp + i16 scale.
`read_wav` reads either int or float WAV payloads and returns f32 in `[-1, 1]`. For int payloads it scales by `1 << (bits_per_sample - 1)`. Unlike a previous version, **per-sample errors are propagated rather than silently dropped** (see watch-out below).
## Data flow
Live capture path (production use):
```
StreamingResampler.push_samples → Vec<f32> 16 kHz mono
└─ WavWriter::append(chunk)
└─ for each f32:
clamp [-1,1] → (i16) write_sample
└─ if samples_since_flush >= 8000:
hound flush (rewrites size headers)
…session ends…
└─ WavWriter::finalize()
└─ terminal header write + close
```
File round-trip (used in tests + simple imports):
```
AudioSamples → write_wav → 16-bit PCM WAV → read_wav → AudioSamples
```
## Watch-outs
- **Crash-safety guarantee is "up to the last flush", not "every sample".** Worst-case loss is `flush_every / sample_rate` seconds (default 500 ms at 16 kHz). The `wav_writer_survives_crash` test (`wav.rs:170`) uses `std::mem::forget` to skip Drop and proves the flushed prefix is recoverable.
- **`finalize` is consuming.** Once called you cannot append further. The live-session command must hold the writer until session end and only finalize at the close path.
- **i16 quantisation noise.** Both writers downcast f32 → i16. WAV round-trip introduces ~1 / 32768 absolute error per sample, validated by the `wav_roundtrip` test (`wav.rs:265`).
- **Truncated WAV bug fixed in tree (2026-04-22 review).** Previous `read_wav` used `filter_map(|s| s.ok())` and silently returned a short `AudioSamples` for corrupt input. Current code surfaces `MagnotiaError::AudioDecodeFailed`. Regression test at `wav.rs:235`.
- **Channels and sample rate are taken from `AudioSamples` directly in `write_wav`.** `WavWriter::create` takes them as explicit args because it precedes the data; the caller (live session) reads `LocalEngine::capabilities()` to know what to pass.
- **No async wrapper.** `WavWriter` operations are synchronous (Buffered file I/O). Live session must call them on a blocking task or accept the syscall cost on the audio thread.
## See also
- [Transcription engines overview](transcription-engines-overview.md) — `LocalEngine::capabilities()` is the source of `sample_rate` / `channels` for `WavWriter::create`.
- [Audio file decoding](audio-file-decoding.md) — `decode_audio_file` is the symphonia-based path used for non-WAV files.
- Tests at `crates/audio/src/wav.rs:165` (crash-safety, finalize round-trip, truncated-input regression).

94
docs/architecture-map/03-audio-transcription/build-tokenizers-guard.md Normal file
View File

@@ -0,0 +1,94 @@
---
name: Build script (Windows tokenizers guard)
type: architecture-map-page
slice: 03-audio-transcription
last_verified: 2026/05/09
---
# Build script (Windows tokenizers guard)
> **Where you are:** [Architecture map](../README.md) → [Audio + Transcription](README.md) → Build script
**Plain English summary.** `magnotia-transcription/build.rs` reads `Cargo.lock` and refuses to compile on Windows if the `tokenizers` crate ever lands in the workspace dependency graph. Linking `whisper-rs-sys` and `tokenizers` together has been a repeated MSVC C-runtime conflict (Whispering v7.11.0 shipped a broken Windows build over exactly this). On non-Windows the check is a `cargo:warning`, not a fatal error, so the failure surfaces at CI build time rather than waiting for a Windows ship.
## At a glance
- Crate: `magnotia-transcription`
- Path: `crates/transcription/build.rs`
- LOC: 73
- External deps: stdlib only (`std::env`, `std::fs`).
- Internal callers: cargo invokes this automatically because `Cargo.toml` declares `build = "build.rs"`.
Public surface: none (build scripts have no callable surface).
## What's in here
```rust
fn main() {
println!("cargo:rerun-if-changed=build.rs");
let target_os = env::var("CARGO_CFG_TARGET_OS").unwrap_or_default();
let manifest_dir = PathBuf::from(env::var("CARGO_MANIFEST_DIR").unwrap_or_else(|_| ".".into()));
// Walk up to workspace root: crates/transcription/ -> crates/ -> root
let workspace_root = manifest_dir
.ancestors()
.find(|p| p.join("Cargo.lock").exists())
.map(PathBuf::from);
let Some(root) = workspace_root else { return; };
let lock_path = root.join("Cargo.lock");
println!("cargo:rerun-if-changed={}", lock_path.display());
let lock = match fs::read_to_string(&lock_path) {
Ok(s) => s,
Err(_) => return,
};
let has_tokenizers = lock
.lines()
.any(|line| matches!(line.trim(), "name = \"tokenizers\""));
if !has_tokenizers { return; }
if target_os == "windows" {
panic!(
"magnotia-transcription: the `tokenizers` crate appears in Cargo.lock and this is a \
Windows build. ..."
);
}
println!(
"cargo:warning=magnotia-transcription: `tokenizers` crate is in the dependency graph. \
This build is non-Windows so the link will succeed, but Windows builds will panic ..."
);
}
```
## Data flow
```
cargo invoke
└─ build.rs
├─ rerun-if-changed=build.rs
├─ ancestors().find(Cargo.lock) → workspace root
├─ rerun-if-changed=Cargo.lock
├─ scan for `name = "tokenizers"`
└─ if found:
target_os == windows? → panic!("brief item #6")
else → cargo:warning
```
## Watch-outs
- **`rerun-if-changed=Cargo.lock` is the trigger.** Adding tokenizers without changing this crate's source still re-runs the script the next build because the lockfile changed.
- **The string match is brittle on purpose.** Looking for the literal `name = "tokenizers"` line in `Cargo.lock` is what TOML pretty-prints. A future cargo version that emits the lockfile differently could miss this. Mitigation: keep the test simple and review on cargo upgrade.
- **The non-Windows path is a warning, not an error.** A CI matrix job on Linux will pass with a yellow message; Windows will fail at build. Fine for a desktop project that ships from Linux first; surprising if anyone ever assumes "Linux green = ready to ship".
- **Brief item #6 reference.** The panic message points at `docs/whisper-ecosystem/brief.md` item #6. Do not lose that pointer in any rewrite — the failure mode is non-obvious without the historical context.
- **`workspace_root` walk falls back gracefully.** If no `Cargo.lock` is found in any ancestor (first-ever cargo run), the script returns early. Subsequent builds will pick up the lock.
- **No way to override.** A maintainer who *wants* to ship `tokenizers` on Windows must delete this `build.rs`, or carry a sidecar process that links tokenizers in its own binary. There is no escape hatch env var.
## See also
- [Transcription whisper](transcription-whisper.md) — `whisper-rs-sys` is the link target this guard protects.
- [Cargo features](cargo-features.md) — same brief-item #6 family of decisions.
- `docs/whisper-ecosystem/brief.md` — the full incident background.

70
docs/architecture-map/03-audio-transcription/cargo-features.md Normal file
View File

@@ -0,0 +1,70 @@
---
name: Cargo feature matrix
type: architecture-map-page
slice: 03-audio-transcription
last_verified: 2026/05/09
---
# Cargo feature matrix
> **Where you are:** [Architecture map](../README.md) → [Audio + Transcription](README.md) → Cargo features
**Plain English summary.** `magnotia-audio` has no Cargo features. `magnotia-transcription` has two: `whisper` (default on, gates `whisper-rs`, `whisper_rs_backend.rs`, and `load_whisper`) and `whisper-vulkan` (default on, additive Vulkan GPU offload). Parakeet is unconditional. The defaults give a desktop GPU build; non-Vulkan or cloud-only configurations turn features off explicitly.
## At a glance
- Crate file: `crates/transcription/Cargo.toml`
- Source of `[features]`: `crates/transcription/Cargo.toml:8-21`.
- Source of conditionals in code: `#[cfg(feature = "whisper")]` in `crates/transcription/src/lib.rs:6` and `:10`, plus the gated `pub mod whisper_rs_backend;` and `load_whisper` re-export. Vulkan check inside `WhisperRsBackend::transcribe_sync` (`crates/transcription/src/whisper_rs_backend.rs:82`).
## Feature table
| Feature | Default | Pulls in | Gates |
|---|---|---|---|
| `whisper` | yes | `whisper-rs 0.16` (optional dep) | `whisper_rs_backend` module, `load_whisper` re-export, the `set_initial_prompt` path |
| `whisper-vulkan` | yes | `whisper-rs?/vulkan` | Compile-time GPU offload. Runtime gated additionally on `vulkan_loader_available()` |
Other deps are unconditional (always present): `transcribe-rs 0.3` (Parakeet), `magnotia-core`, `tokio`, `reqwest`, `futures-util`, `sha2`, `thiserror`, `tracing`. Dev-only: `tempfile`, `num_cpus` (used by `thread_sweep.rs` to label physical vs logical core counts).
## Build commands
Default (Whisper + Vulkan):
```sh
cargo build -p magnotia-transcription
```
Whisper without Vulkan (CPU-only desktop, Android without GPU drivers):
```sh
cargo build -p magnotia-transcription --no-default-features --features whisper
```
No Whisper at all (Parakeet only — useful for shrinking the binary and dropping `whisper-rs-sys`):
```sh
cargo build -p magnotia-transcription --no-default-features
```
The `--no-default-features` form drops the `whisper_rs_backend` module entirely, so any sibling code holding a `WhisperRsBackend` will fail to compile. `LocalEngine`, `SpeechModelAdapter`, `load_parakeet`, and the streaming primitives stay available.
## Rationale (from Cargo.toml comments)
- `whisper` exists as a feature so a future Windows non-AVX2 build, or a cloud-only ASR config, can drop `whisper-rs-sys` entirely (brief item #13). Disabling it also drops the `WhisperRsBackend` module and the `load_whisper` entry point.
- `whisper-vulkan` is a separate feature so a non-Vulkan target (Android without GPU drivers, a CPU-only Windows build) can pull in `whisper-rs` but skip the Vulkan backend.
- `transcribe-rs` is unconditional because today's only second engine (Parakeet) goes through it. Brief item #13's hypothetical "drop both engines" case would need a second feature flag introduced here.
## Watch-outs
- **`whisper-vulkan` is purely additive.** Disabling it does not change `whisper`'s default build except for skipping the Vulkan backend selection. There is no separate "force CPU" or "force GPU" runtime flag; the backend probes `vulkan_loader_available()` and uses Vulkan if present.
- **`thiserror` and `tracing` are not feature-gated.** Comments justify keeping them unconditional (small cost, broad usage).
- **Reqwest features are pinned.** `default-features = false, features = ["rustls-tls", "stream"]`. There is no `native-tls` fallback path; `--features reqwest/native-tls` would rebuild the dep with the wrong TLS backend on Linux.
- **`whisper-rs` is `default-features = false, optional = true`.** Whisper-rs's own default features can re-enable backends we don't want; setting `default-features = false` keeps the dep minimal and lets `whisper-vulkan` add the one feature we care about.
- **Switching feature sets between builds invalidates the build cache.** A common slow-down for first-time contributors switching `--no-default-features` builds.
## See also
- [Transcription whisper](transcription-whisper.md) — the gated module.
- [Transcription parakeet](transcription-parakeet.md) — note Parakeet has no feature flag.
- [Build tokenizers guard](build-tokenizers-guard.md) — Windows MSVC defence is part of the same brief item #6 thinking.
- `crates/transcription/Cargo.toml` — the source of truth.

86
docs/architecture-map/03-audio-transcription/tests-and-fixtures.md Normal file
View File

@@ -0,0 +1,86 @@
---
name: Tests and fixtures
type: architecture-map-page
slice: 03-audio-transcription
last_verified: 2026/05/09
---
# Tests and fixtures
> **Where you are:** [Architecture map](../README.md) → [Audio + Transcription](README.md) → Tests and fixtures
**Plain English summary.** This slice has three categories of test. **Inline unit tests** under `#[cfg(test)] mod tests` exercise every public type (capture monitor detection, resampler duration preservation, RmsVadChunker state machine, LocalAgreement invariants, buffer trim bounds, model-manager hash and resume paths). **Integration tests** under `crates/transcription/tests/` exercise whisper-rs end-to-end with env-var-gated fixtures so CI without a model checkpoint exits quietly. **Test-only fixtures** include in-tree `tokio::net::TcpListener` HTTP servers that simulate Range / 5xx / no-Range-honour servers for the download stack.
## At a glance
- Crate: `magnotia-transcription` (mostly) and `magnotia-audio` (inline only).
- Paths:
- `crates/audio/src/{capture,decode,wav,resample,streaming_resample}.rs` — inline `#[cfg(test)]` modules.
- `crates/transcription/src/{model_manager,local_engine,transcriber,streaming/*}.rs` — inline `#[cfg(test)]` modules.
- `crates/transcription/tests/whisper_rs_smoke.rs` — env-gated load + transcribe.
- `crates/transcription/tests/jfk_bench.rs` — env-gated benchmark with cold/warm RTF.
- `crates/transcription/tests/thread_sweep.rs` — env-gated thread-count scaling sweep.
- External deps (dev): `tempfile 3`, `tokio` (rt, sync, net, io-util, macros), `num_cpus 1` (label-only).
## Inline unit tests (selected highlights)
### `magnotia-audio`
- `capture.rs:570` — `monitor_pattern_detection` for the PulseAudio / PipeWire heuristics.
- `decode.rs:262` — `mid_stream_io_error_propagates_instead_of_returning_partial_audio`. RB-09 regression: a flaky `MediaSource` (`FlakyCursor`, `decode.rs:203`) injects an I/O error after the probe header and asserts the function errors instead of returning `Ok(partial)`.
- `wav.rs:170` — `wav_writer_survives_crash`. Uses `std::mem::forget` to skip the hound finaliser, simulating a process kill, then asserts the flushed prefix is recoverable via `read_wav`.
- `wav.rs:235` — `read_wav_surfaces_truncated_sample_stream_errors`. 2026-04-22 regression: the previous `filter_map(|s| s.ok())` swallowed errors silently.
- `streaming_resample.rs:182` — `streaming_48k_to_16k_preserves_duration`. Pushes 1 s of 48 kHz audio in 700-sample chunks (forcing the residual-buffer path) and asserts the output is within 50 ms of 1 s at 16 kHz.
- `resample.rs:84` — `resample_preserves_approximate_duration`.
### `magnotia-transcription`
- `streaming/rms_vad.rs:371`–`:734` — comprehensive state-machine coverage: silence, hysteresis, onset gating, max-chunk continuity (`max_chunk_split_preserves_audio_contiguity`), padded-final-frame regression (`flush_preserves_hit_max_chunk_from_padded_final_frame`, `flush_preserves_end_of_utterance_chunk_from_padded_final_frame` — both 2026-04-22 CRITICAL C2 fixes), idempotent flush.
- `streaming/commit_policy.rs:212`–`:402` — LocalAgreement-n correctness: text-only equality, non-shrinkage invariant, defensive shorter-pass handling (`shorter_pass_after_commit_does_not_panic`, regression for the `latest[committed_count..]` panic), n=3 behaviour.
- `streaming/buffer_trim.rs:56`–`:206` — defensive non-finite handling, integration test against `LocalAgreement::last_committed_end_secs`, the long-session bound proof (`trim_bounds_buffer_over_long_session`).
- `model_manager.rs:339`–`:615` — three TcpListener-backed fixtures (`spawn_range_server`, `spawn_no_range_server`, `spawn_500_server`) drive `download_file` through the resume path, the ResumeUnsupported path (server returns 200 to a Range request), the 5xx rejection path, and the SHA-mismatch cleanup path.
- `local_engine.rs:218` — `engine_reports_not_available_before_loading`.
- `transcriber.rs:54` — `transcriber_trait_is_object_safe` (compile-time witness).
## Env-gated integration tests (`crates/transcription/tests/`)
All three skip silently if the env vars are unset, so `cargo test` on a fresh machine is a no-op for them.
### `whisper_rs_smoke.rs`
- Env vars: `MAGNOTIA_WHISPER_TEST_MODEL` (path to a ggml/gguf Whisper model).
- Loads the context, builds a state, calls `set_initial_prompt("Wren, CORBEL, ADHD")` (proves the API still exists), runs inference on 1 s of silence, exercises `full_n_segments` and `get_segment(i).to_str()`. Smoke test only; assertion is "did this run without panicking".
### `jfk_bench.rs`
- Env vars: `MAGNOTIA_WHISPER_TEST_MODEL`, `MAGNOTIA_WHISPER_TEST_AUDIO` (path to a 16 kHz mono 16-bit PCM WAV — assertions in the test enforce this format on the fixture, not the runtime).
- Reports cold-load time, cold-transcribe RTF, warm-transcribe RTF, peak RSS read from `/proc/{pid}/status`. Hardcodes `set_n_threads(6)` for both runs (so a sweep across thread counts is a separate test, not this one).
### `thread_sweep.rs`
- Same env vars as `jfk_bench.rs`. Adds `MAGNOTIA_POWER_STATE_OVERRIDE` (set internally per panel to `ac` or `battery`) so `inference_thread_count` returns its predicted pick for each combination of (power state × GPU offload) and the empirical RTF table can be compared against the helper's choice.
- Runs the JFK clip at `n_threads = 1, 2, 4, physical, logical` (plus `8` if logical ≥ 8), takes the min of two runs per setting, prints a four-panel table:
1. AC, CPU.
2. AC, GPU (Vulkan).
3. battery, CPU.
4. battery, GPU (Vulkan).
- Uses `num_cpus::get()` and `num_cpus::get_physical()` purely for the printed labels (`thread_sweep.rs:35`). The runtime helper `inference_thread_count(Workload::Whisper, gpu_offloaded)` does the actual prediction.
- `vulkan_runtime_ok` snapshot taken once via `cfg!(feature = "whisper-vulkan") && vulkan_loader_available()`.
## Watch-outs
- **All three integration tests are quiet on missing env vars.** A green `cargo test` does NOT mean the Whisper path works; it means "either the path works or you didn't set the env vars". CI configurations must explicitly set `MAGNOTIA_WHISPER_TEST_MODEL` to exercise these.
- **`jfk_bench.rs` hardcodes `n_threads = 6`.** Production code uses `inference_thread_count(Workload::Whisper, gpu_offloaded)` which is power-aware. Bench numbers are not directly comparable to runtime numbers without rerunning with the helper-picked thread count.
- **`thread_sweep.rs` mutates `MAGNOTIA_POWER_STATE_OVERRIDE` via `env::set_var`.** Globally process-wide. Concurrent tests in the same `cargo test` invocation that read `MAGNOTIA_POWER_STATE_OVERRIDE` will race. The test removes the var at the end (`thread_sweep.rs:94`).
- **`tempfile 3` is a dev-dep, not a runtime dep.** Production code never creates temp files.
- **`num_cpus = "1"` (the version, not "one CPU").** Test-only labelling; production uses the slice 5 helper.
- **In-tree HTTP fixtures use ephemeral ports (`bind 127.0.0.1:0`) and run on a tokio task.** No port-collision risk, but a paranoid sandbox that blocks raw TCP loopback would break these tests.
- **`ModelFile`'s `&'static str` fields force the test helper `leak()` (`model_manager.rs:462`).** Each test leaks a few small strings. Acceptable for one-shot tests; do not copy this pattern into production code.
## See also
- [Transcription model manager](transcription-model-manager.md) — the production code these download tests guard.
- [Transcription whisper](transcription-whisper.md) — the production code these integration tests guard.
- [Transcription streaming](transcription-streaming.md) — exhaustive unit-test coverage lives alongside this code.
- `docs/whisper-ecosystem/brief.md` — the source of the items (`#6, #8, #19, #21, #24, #25`) repeatedly referenced in test docstrings.
- `docs/code-review-2026-04-22.md` — the audit pass that produced the C2, RB-09, and `read_wav` regressions captured here.

67
docs/architecture-map/03-audio-transcription/transcription-concurrency.md Normal file
View File

@@ -0,0 +1,67 @@
---
name: Inference concurrency (run_inference)
type: architecture-map-page
slice: 03-audio-transcription
last_verified: 2026/05/09
---
# Inference concurrency
> **Where you are:** [Architecture map](../README.md) → [Audio + Transcription](README.md) → Inference concurrency
**Plain English summary.** Whisper and Parakeet inference is synchronous and CPU-bound (or GPU-bound through Vulkan). Running it on the Tokio runtime would block the executor, so every async caller goes through `run_inference`, a thin wrapper that hands the work to `tokio::task::spawn_blocking`. Callers never see the `Arc<LocalEngine>` lock or the spawn boundary; they `await` a `TimedTranscript`.
## At a glance
- Crate: `magnotia-transcription`
- Path: `crates/transcription/src/concurrency.rs`
- LOC: 19
- External deps: `tokio` (rt feature for `spawn_blocking`).
- Internal callers (best effort, slice 2 reconciles): `src-tauri/src/commands/transcription.rs` (file path), `src-tauri/src/commands/live.rs` (per-chunk during live capture).
Public surface:
- `pub async fn run_inference(engine: Arc<LocalEngine>, audio: AudioSamples, options: TranscriptionOptions) -> Result<TimedTranscript>` — `crates/transcription/src/concurrency.rs:11`.
## What's in here
The whole file:
```rust
pub async fn run_inference(
engine: Arc<LocalEngine>,
audio: AudioSamples,
options: TranscriptionOptions,
) -> Result<TimedTranscript> {
tokio::task::spawn_blocking(move || engine.transcribe_sync(&audio, &options))
.await
.map_err(|e| MagnotiaError::TranscriptionFailed(format!("Task join error: {e}")))?
}
```
`engine` is shared via `Arc` (the engine itself uses an internal mutex for inference / load serialisation). `audio` and `options` are moved into the closure. The double `?` flattens the join-error layer with the inference-error layer.
## Data flow
```
async caller
└─ run_inference(engine, audio, options)
└─ tokio::task::spawn_blocking(move || engine.transcribe_sync(...))
└─ LocalEngine internal mutex → Transcriber::transcribe_sync
→ Vec<Segment> + inference_ms
→ Result<TimedTranscript>
.await + map_err(JoinError → MagnotiaError::TranscriptionFailed)
```
## Watch-outs
- **`spawn_blocking` is the only place inference runs.** Anyone writing a new caller that calls `LocalEngine::transcribe_sync` directly from an async context will block the Tokio executor. Always go through `run_inference`.
- **`Arc<LocalEngine>` is the contract.** The function takes an `Arc`, not `&LocalEngine`, because the engine must outlive the spawned task. The shared state in slice 2 holds the engine inside an `Arc`.
- **Audio and options are moved.** They are not borrowed across the spawn boundary; the caller hands ownership over.
- **JoinError is the only failure mode added by this layer.** `MagnotiaError::TranscriptionFailed("Task join error: ...")` indicates a panic inside `transcribe_sync` or a runtime shutdown. The actual inference errors flow through the inner `Result`.
- **No backpressure here.** Callers issuing many concurrent `run_inference` calls each spawn an independent blocking task. Tokio's default blocking thread pool is 512 threads, but the `LocalEngine` mutex serialises them — only one will actually run inference at a time. Worth knowing if you ever look at thread counts in a profiler.
## See also
- [Transcription engines overview](transcription-engines-overview.md) — `LocalEngine::transcribe_sync` is what runs inside the blocking task.
- [Transcription streaming](transcription-streaming.md) — live capture invokes `run_inference` per chunk.

107
docs/architecture-map/03-audio-transcription/transcription-engines-overview.md Normal file
View File

@@ -0,0 +1,107 @@
---
name: Transcription engines overview
type: architecture-map-page
slice: 03-audio-transcription
last_verified: 2026/05/09
---
# Transcription engines overview
> **Where you are:** [Architecture map](../README.md) → [Audio + Transcription](README.md) → Transcription engines overview
**Plain English summary.** Every speech-to-text backend implements a single `Transcriber` trait. `LocalEngine` owns the currently-loaded backend behind a `Mutex`, exposes `transcribe_sync` for inference, and `load` / `unload` so the sequential-GPU guard can free VRAM before the LLM loads. Two concrete `Transcriber` implementations exist: `WhisperRsBackend` (direct `whisper-rs`, the only one that pipes `initial_prompt`) and `SpeechModelAdapter` (wraps any `transcribe-rs` `SpeechModel`, used today for Parakeet via a thin `ParakeetWordGranularity` shim).
## At a glance
- Crate: `magnotia-transcription`
- Paths: `crates/transcription/src/transcriber.rs` (62 LOC), `crates/transcription/src/local_engine.rs` (226 LOC).
- External deps: `transcribe-rs 0.3` (onnx feature). `whisper-rs 0.16` is feature-gated and lives in the next page.
- Internal callers (best effort, slice 2 reconciles): `src-tauri/src/commands/models.rs` calls `load_whisper` / `load_parakeet` and `LocalEngine::load`. `src-tauri/src/commands/transcription.rs` (and `live.rs`) call `LocalEngine::transcribe_sync` indirectly via `concurrency::run_inference`.
Public surface:
- `pub trait Transcriber: Send` — `crates/transcription/src/transcriber.rs:35`. Methods: `capabilities()`, `transcribe_sync(&mut self, samples, options)`.
- `pub struct TranscriberCapabilities` — `crates/transcription/src/transcriber.rs:23`. Fields: `sample_rate: u32`, `channels: u16`, `supports_initial_prompt: bool`.
- `pub struct LocalEngine` — `crates/transcription/src/local_engine.rs:68`.
- `pub struct SpeechModelAdapter(pub Box<dyn SpeechModel + Send>)` — `crates/transcription/src/local_engine.rs:26`.
- `pub struct TimedTranscript { transcript: Transcript, inference_ms: u64 }` — `crates/transcription/src/local_engine.rs:17`.
- `pub fn load_parakeet(model_dir: &Path) -> Result<Box<dyn Transcriber + Send>>` — `crates/transcription/src/local_engine.rs:197`.
- `pub fn load_whisper(model_path: &Path) -> Result<Box<dyn Transcriber + Send>>` — `crates/transcription/src/local_engine.rs:208` (gated behind `feature = "whisper"`).
- `pub use transcribe_rs::SpeechModel` — re-exported via `lib.rs:18`.
## What's in here
### `Transcriber` trait
`transcriber.rs:35`. `Send` is a supertrait so `Box<dyn Transcriber + Send>` travels through `tokio::task::spawn_blocking` boundaries. Synchronous-only API: async callers wrap a `spawn_blocking`. Inference is `&mut self` so backends with per-call scratch state (whisper-rs `WhisperState`, Parakeet decoder buffers) can mutate without interior-mutability gymnastics.
`TranscriberCapabilities` carries three fields:
- `sample_rate: u32` — the rate this backend wants. Live capture's `WavWriter::create` reads this to decide how to format the on-disk file (brief item #19).
- `channels: u16` — almost always 1, kept explicit so a stereo-aware backend can declare it.
- `supports_initial_prompt: bool` — Whisper says yes, Parakeet says no. Settings UI hides the field accordingly.
The trait is asserted object-safe by a compile-time witness test (`transcriber.rs:54`).
### `LocalEngine`
`local_engine.rs:68`. Holds:
- `engine: Mutex<Option<Box<dyn Transcriber + Send>>>` — currently-loaded backend or `None`.
- `engine_name: EngineName` — display tag (e.g. "whisper", "parakeet"); set at construction.
- `loaded_model_id: Mutex<Option<ModelId>>` — what's loaded. Used by the UI and by the model-switch logic.
API:
- `new(engine_name)` — create empty.
- `load(backend, model_id)` — install a backend; replaces any prior.
- `unload()` — drop the backend so its GPU VRAM / mmap'd tensors are freed (sequential-GPU guard, brief item A.1 #28).
- `name() -> &EngineName`.
- `loaded_model_id() -> Option<ModelId>`.
- `is_loaded() -> bool`.
- `capabilities() -> Option<TranscriberCapabilities>`.
- `transcribe_sync(audio, options) -> Result<TimedTranscript>` — locks the engine mutex, invokes the trait, times it with `Instant::now`, wraps the segments in `Transcript::new(segments, language, duration_secs)`.
The mutex serialises inference against load / unload. There is no per-call lock contention with the audio thread; all inference is ferried through `concurrency::run_inference` (see [transcription-concurrency.md](transcription-concurrency.md)).
### `SpeechModelAdapter`
`local_engine.rs:26`. Adapter from any `transcribe-rs` `SpeechModel` to the `Transcriber` trait. Implements `capabilities()` returning `sample_rate = WHISPER_SAMPLE_RATE`, `channels = 1`, `supports_initial_prompt = false`. `transcribe_sync` constructs a `TranscribeOptions { language, translate: false, leading_silence_ms: None, trailing_silence_ms: None }` and converts the resulting `TranscriptionResult.segments` into `Vec<Segment>`.
### Loaders
- `load_whisper` (gated behind `feature = "whisper"`) constructs a `WhisperRsBackend` (see next page).
- `load_parakeet` constructs a `ParakeetModel` with `Quantization::Int8`, wraps it in `ParakeetWordGranularity` (so the trait method requests `TimestampGranularity::Word` instead of the per-subword default), then wraps that in `SpeechModelAdapter`.
## Data flow
```
audio: AudioSamples (16 kHz mono)
options: TranscriptionOptions (language, initial_prompt)
└─ run_inference (concurrency.rs)
└─ spawn_blocking
└─ LocalEngine::transcribe_sync
├─ Mutex::lock(engine)
├─ backend.transcribe_sync(samples, options)
│ └─ WhisperRsBackend or SpeechModelAdapter
├─ Instant::now diff → inference_ms
└─ Transcript::new(segments, language, duration_secs)
→ TimedTranscript
```
## Watch-outs
- **`engine_name` is not validated.** `LocalEngine::new(EngineName::new("whisper"))` is opaque to the engine itself. Anything that conditionally branches on engine name (e.g. UI selecting the right model list) must agree on the string value with the loader and Tauri command.
- **`Mutex::lock().unwrap_or_else(|e| e.into_inner())` is used everywhere.** A poisoned mutex (panicking call) does not propagate — the engine is read in its post-panic state. Acceptable for a single-process desktop app; document this if anyone moves the type into a daemon.
- **`Transcript::new` requires the audio duration.** Computed via `audio.duration_secs()`; assumes `AudioSamples` carries the original sample count and rate (it does). If anyone ever passes a pre-resampled vector that has lost the rate field, the duration will be wrong.
- **No streaming surface in this trait.** All inference is one-shot synchronous. Live capture turns this into a streaming experience by chunking the audio (see `RmsVadChunker`) and stitching segments via `LocalAgreement`.
- **Parakeet wrapper hardcodes `Word` granularity.** `ParakeetWordGranularity::transcribe_raw` (`local_engine.rs:182`) overrides the default `Token` granularity that produced "T Est Ing , One" output. If a future caller wants character-level timestamps, this is where they'd need a flag.
- **Adapter discards `initial_prompt`.** `SpeechModelAdapter::transcribe_sync` does not look at `options.initial_prompt`. Whisper is the only path that pipes it; the capability flag exists so the UI does not pretend otherwise.
## See also
- [Transcription whisper](transcription-whisper.md) — the whisper-rs backend.
- [Transcription parakeet](transcription-parakeet.md) — the transcribe-rs Parakeet wrapper.
- [Transcription concurrency](transcription-concurrency.md) — `run_inference` async wrapper.
- [Transcription streaming](transcription-streaming.md) — VAD chunker + LocalAgreement stack that drives live capture.
- `magnotia_core::types::{Segment, Transcript, TranscriptionOptions, EngineName, ModelId}` (slice 5).

132
docs/architecture-map/03-audio-transcription/transcription-model-manager.md Normal file
View File

@@ -0,0 +1,132 @@
---
name: Model manager (download, verify, locate)
type: architecture-map-page
slice: 03-audio-transcription
last_verified: 2026/05/09
---
# Model manager (download, verify, locate)
> **Where you are:** [Architecture map](../README.md) → [Audio + Transcription](README.md) → Model manager
**Plain English summary.** Speech models live on disk under a per-platform path. `model_manager` resolves those paths, checks "is this model fully downloaded", lists what is downloaded, and downloads missing files with HTTP Range resume + per-chunk SHA256 verification. A single in-process `HashSet` reservation prevents concurrent downloads of the same model. After a successful download a `.magnotia-verified` manifest is written so a subsequent process restart can short-circuit re-hashing.
## At a glance
- Crate: `magnotia-transcription`
- Path: `crates/transcription/src/model_manager.rs`
- LOC: 617 (production code + extensive in-tree download server tests)
- External deps: `reqwest 0.12` (rustls-tls, stream), `futures-util 0.3` (StreamExt), `sha2 0.10`. Path resolution comes from `magnotia_core::paths::app_paths`. Catalogue from `magnotia_core::model_registry`.
- Internal callers (best effort, slice 2 reconciles): the `models` Tauri command consumes all five public functions. The frontend Settings → Models view drives `download` with a progress callback.
Public surface:
- `pub fn models_dir() -> PathBuf` — `crates/transcription/src/model_manager.rs:42`. Resolves to `%LOCALAPPDATA%/magnotia/models` on Windows, `~/.magnotia/models` on Unix (delegates to `magnotia_core::paths::app_paths`).
- `pub fn model_dir(id: &ModelId) -> PathBuf` — `crates/transcription/src/model_manager.rs:47`.
- `pub fn is_downloaded(id: &ModelId) -> bool` — `crates/transcription/src/model_manager.rs:52`.
- `pub fn list_downloaded() -> Vec<ModelId>` — `crates/transcription/src/model_manager.rs:63`.
- `pub async fn download(id: &ModelId, progress: impl Fn(DownloadProgress) + Send + 'static) -> Result<()>` — `crates/transcription/src/model_manager.rs:78`.
Private helpers:
- `DownloadReservation` — RAII guard (`model_manager.rs:12`).
- `verified_manifest_path` / `verified_manifest_matches` / `write_verified_manifest` (`model_manager.rs:115`–`:153`).
- `sha256_of_file` (`model_manager.rs:157`).
- `download_file` (`model_manager.rs:178`).
## What's in here
### Path resolution
`models_dir` and `model_dir` defer entirely to `magnotia_core::paths::app_paths()`. Anything platform-specific lives in slice 5; this crate just consumes the resolution.
### Download reservation
`DownloadReservation::acquire(id)` (`model_manager.rs:17`) inserts the model id into a process-wide `HashSet` guarded by a `LazyLock<Mutex<...>>`. Returns `Err(MagnotiaError::DownloadFailed("download already in progress for {id}"))` if the id is already present. Drop removes the entry. This protects against re-entry from the same process; cross-process races are still possible but the atomic `.part → dest` rename closes the most damaging window.
### `is_downloaded`
`model_manager.rs:52`. Two-step:
1. Every file declared in the registry entry must exist at `model_dir(id).join(filename)`.
2. The verified-manifest at `model_dir.join(".magnotia-verified")` must record the same `filename\tsha256\tsize` triple per file.
A missing or stale manifest causes `is_downloaded` to return `false` even if the bytes are correct. The next `download` call will revalidate and rewrite the manifest. A truncated or tampered file fails the existence + size check, so a subsequent `download` redownloads it.
### `list_downloaded`
`model_manager.rs:63`. Filters the static `magnotia_core::model_registry::all_models()` catalogue by `is_downloaded`.
### `download` (the orchestrator)
`model_manager.rs:78`. For each `ModelFile` in the registry entry:
1. If `dest` exists, hash it with `sha256_of_file`. Match → skip the download. Mismatch → delete and re-fetch. Hash IO error → propagate as `MagnotiaError::DownloadFailed`.
2. Otherwise call `download_file(file, dest, id, &progress)`.
After all files are present and verified, `write_verified_manifest` writes the `version\t1` header and per-file `filename\tsha256\tsize` lines.
### `download_file` (single file with resume + hash)
`model_manager.rs:178`. Implements the Buzz-style resume pattern:
1. Build a `reqwest::Client` with a 30 s connect timeout.
2. If `dest.with_extension("part")` exists, send a `Range: bytes={existing}-` header.
3. Inspect the response status:
- `206 Partial Content` → server is resuming. Continue appending.
- `200 OK` after a Range request → server ignored Range. Discard the stale `.part` and start fresh (regression-tested at `model_manager.rs:494`).
- `4xx`/`5xx` (non-resume path) → `MagnotiaError::DownloadFailed("download returned HTTP {status} for {filename}")`. Crucial because `reqwest` does not auto-error on bad status; without this check, a 404 body would be streamed into `.part` and renamed over the destination (regression at `model_manager.rs:556`).
- Other status on resume → propagate as unexpected.
4. Determine `total_bytes` from `Content-Range` (resume) or `Content-Length` (fresh).
5. Hasher state: when resuming, hash the on-disk `.part` first to catch the hash up to where the network stream is about to resume. Otherwise start fresh.
6. Stream the body in chunks, writing each to the file, updating the running SHA256, emitting `DownloadProgress` per percent step.
7. On stream end, finalise the hash. Mismatch → remove `.part` and `MagnotiaError::DownloadFailed("SHA256 mismatch for {filename}: expected ..., got ...")`.
8. `std::fs::rename(.part, dest)` — atomic finalise.
### `sha256_of_file`
`model_manager.rs:157`. 8 KiB buffer streaming hash. Used by `download` to validate an existing complete file before trusting it.
## Data flow
```
ModelId
└─ models_dir / model_dir → on-disk directory
└─ is_downloaded:
all files exist AND verified-manifest matches
→ true / false
└─ list_downloaded:
filter all_models() by is_downloaded
└─ download:
acquire reservation
for file in entry.files:
if dest exists:
sha256_of_file → match? skip : delete + re-fetch
else:
download_file:
[.part exists? Range request : full GET]
stream body → file + hasher + progress callback
match SHA256 → atomic rename
else → cleanup, error
write_verified_manifest
drop reservation
```
## Watch-outs
- **Reservation is per-process.** Two Magnotia instances (dev + release, two cargo runs) racing on the same model dir can collide. The `.part → dest` atomic rename is the only thing standing between them and a torn file.
- **`DownloadProgress` is emitted per percent step, not per byte.** A 10 MB file at 4G download speeds emits ~100 events; a 5 GB file emits the same 100 events. UI must not assume tight cadence.
- **Manifest schema is `version\t1` + `filename\tsha256\tsize` lines.** Bumping the version is a manual operation; a stale manifest after a registry update will simply trigger a redownload (waste, not corruption).
- **Cross-process file locking is not used.** A user opening Settings → Models in two windows of the same desktop session will trip the in-process reservation, but two physical machines syncing the same network home would not be detected.
- **Reqwest connect timeout is 30 s, no read timeout.** A stalled mid-stream connection blocks indefinitely. Acceptable for a foreground UI flow today; would need a `tokio::time::timeout` wrap if the download ever became headless.
- **`ModelFile::sha256` is `&'static str`.** The `leak` helper in tests reflects that production code reads a static catalogue compiled into the binary. Tampering with the catalogue requires a code change, not a runtime config.
- **`reqwest` is configured `default-features = false, features = ["rustls-tls", "stream"]`.** No native-tls fallback. Linux without ca-certificates will fail at connect.
- **`models_dir` is the user-data dir, not a cache dir.** A "clear cache" button in the OS would not nuke models. Documented behaviour.
## See also
- [Transcription engines overview](transcription-engines-overview.md) — `load_whisper` / `load_parakeet` are called against `model_dir(id)` outputs.
- [Tests and fixtures](tests-and-fixtures.md) — in-tree TcpListener-backed servers that exercise the resume + sha-mismatch + 5xx paths.
- `magnotia_core::model_registry::{find_model, all_models, ModelEntry, ModelFile}` (slice 5) — the source of URLs and hashes.
- `magnotia_core::paths::app_paths` (slice 5) — platform path resolution.
- `magnotia_core::types::DownloadProgress` (slice 5) — the progress event shape.

89
docs/architecture-map/03-audio-transcription/transcription-parakeet.md Normal file
View File

@@ -0,0 +1,89 @@
---
name: Parakeet backend (transcribe-rs ONNX)
type: architecture-map-page
slice: 03-audio-transcription
last_verified: 2026/05/09
---
# Parakeet backend (transcribe-rs ONNX)
> **Where you are:** [Architecture map](../README.md) → [Audio + Transcription](README.md) → Parakeet backend
**Plain English summary.** Parakeet is loaded via `transcribe-rs` 0.3 with the `onnx` feature. The model itself is wrapped in a thin `ParakeetWordGranularity` shim that overrides the trait method to request word-level timestamps (instead of the default per-subword "T Est Ing" output). The shim is then wrapped in `SpeechModelAdapter` so the rest of the engine sees a uniform `Transcriber`. There is no Cargo feature for Parakeet today: the dep is unconditional.
## At a glance
- Crate: `magnotia-transcription`
- Path: `crates/transcription/src/local_engine.rs:167` (the `ParakeetWordGranularity` shim) and `:197` (`load_parakeet`).
- LOC (Parakeet-specific code): ~40.
- External deps: `transcribe-rs 0.3` with `default-features = false, features = ["onnx"]`.
- Internal callers (best effort, slice 2 reconciles): `src-tauri/src/commands/models.rs` calls `load_parakeet` from the model-load command path.
Public surface:
- `pub fn load_parakeet(model_dir: &Path) -> Result<Box<dyn Transcriber + Send>>` — `crates/transcription/src/local_engine.rs:197`.
- `pub use transcribe_rs::SpeechModel` — re-exported in `lib.rs:18` for any caller that wants to hold a raw `SpeechModel`.
## What's in here
### `ParakeetWordGranularity` (private struct)
`local_engine.rs:167`. Wraps `transcribe_rs::onnx::parakeet::ParakeetModel`. Implements `transcribe_rs::SpeechModel` by forwarding `capabilities()`, `default_leading_silence_ms()`, `default_trailing_silence_ms()` to the inner model, but overriding `transcribe_raw`:
```rust
fn transcribe_raw(&mut self, samples, options) -> ...TranscriptionResult... {
use transcribe_rs::onnx::parakeet::{ParakeetParams, TimestampGranularity};
let params = ParakeetParams {
language: options.language.clone(),
timestamp_granularity: Some(TimestampGranularity::Word),
};
self.0.transcribe_with(samples, &params)
}
```
Why this exists: `transcribe-rs` 0.3's blanket `SpeechModel` impl for `ParakeetModel::transcribe_raw` ignores `TranscribeOptions` and uses `TimestampGranularity::Token` (per-subword), which surfaces in Magnotia as `T Est Ing . One , Two , Three` output. The concrete-type method `ParakeetModel::transcribe_with` accepts `ParakeetParams` with an explicit granularity. The shim exposes that to the trait object.
### `load_parakeet`
`local_engine.rs:197`. `ParakeetModel::load(model_dir, &Quantization::Int8)` then wraps:
```rust
SpeechModelAdapter(Box::new(ParakeetWordGranularity(model)))
```
Hardcodes `Quantization::Int8`. No feature flag; no caller-side override.
## Data flow
```
samples (f32, 16 kHz mono)
options: TranscriptionOptions { language, initial_prompt } // initial_prompt discarded
└─ SpeechModelAdapter::transcribe_sync
└─ TranscribeOptions {
language,
translate: false,
leading_silence_ms: None,
trailing_silence_ms: None,
}
└─ ParakeetWordGranularity::transcribe_raw
└─ ParakeetParams { language, timestamp_granularity: Word }
└─ ParakeetModel::transcribe_with
└─ TranscriptionResult { segments: Option<Vec<...>> }
→ Vec<Segment> (start_secs, end_secs as f64; text)
```
## Watch-outs
- **`initial_prompt` is silently discarded.** `SpeechModelAdapter::transcribe_sync` does not look at it. The capability flag on `TranscriberCapabilities { supports_initial_prompt: false }` is what tells the UI to hide the field. Don't let a UI refactor accidentally show it.
- **Quantisation is locked to Int8.** No way to pick FP16 or FP32 today without editing `load_parakeet`. May matter for accuracy / VRAM trade-offs on bigger Parakeet variants.
- **Segment timestamps already come back in seconds.** Unlike whisper-rs's centiseconds, `transcribe-rs` returns floats already in seconds. The `as f64` casts in `SpeechModelAdapter` are widening, not unit conversion.
- **No Cargo feature gate.** `transcribe-rs` is always pulled in. Disabling Parakeet would require either a new feature in this crate, or editing `lib.rs` re-exports. Brief item #13 hints a future cloud-only ASR config might want to drop both backends; today only Whisper is gateable.
- **`ParakeetParams` differ between transcribe-rs versions.** If you bump `transcribe-rs`, `transcribe_with` and `ParakeetParams` are the API surface to re-verify. The shim is the only file that constructs `ParakeetParams` directly.
- **`ort` version conflict is the reason Silero VAD is blocked.** Parakeet (transcribe-rs) requires `ort 2.0.0-rc.12`; the Silero crates pin `2.0.0-rc.10`. Documented in [audio-vad.md](audio-vad.md).
## See also
- [Transcription engines overview](transcription-engines-overview.md) — the `Transcriber` / `SpeechModelAdapter` plumbing.
- [Audio VAD](audio-vad.md) — the ort version conflict introduced by this dep.
- [Cargo features](cargo-features.md) — note Parakeet has no feature flag today.
- `transcribe_rs::SpeechModel` re-export at `crates/transcription/src/lib.rs:18`.

146
docs/architecture-map/03-audio-transcription/transcription-streaming.md Normal file
View File

@@ -0,0 +1,146 @@
---
name: Streaming primitives (VAD chunker, LocalAgreement, buffer trim)
type: architecture-map-page
slice: 03-audio-transcription
last_verified: 2026/05/09
---
# Streaming primitives
> **Where you are:** [Architecture map](../README.md) → [Audio + Transcription](README.md) → Streaming primitives
**Plain English summary.** Live capture cannot just feed Whisper a single 30-minute buffer at session end. Three primitives cooperate to produce responsive live transcription: `RmsVadChunker` decides when "speech" has ended and emits a `VadChunk`; `LocalAgreement` holds tokens as tentative until two consecutive Whisper passes agree on a prefix, then commits that prefix; `trim_buffer_to_commit_point` drains committed audio out of the working buffer so memory stays bounded. The trait surface (`VadChunker`) matches what a future Silero implementation will provide, so when the `ort` version conflict resolves, the live session keeps working.
## At a glance
- Crate: `magnotia-transcription`
- Paths:
- `crates/transcription/src/streaming/mod.rs` (84 LOC) — trait + types + re-exports.
- `crates/transcription/src/streaming/rms_vad.rs` (736 LOC) — `RmsVadChunker`.
- `crates/transcription/src/streaming/commit_policy.rs` (404 LOC) — `LocalAgreement`, `Token`, `CommitDecision`, `CommitPolicy`.
- `crates/transcription/src/streaming/buffer_trim.rs` (208 LOC) — `sample_index_for_seconds`, `trim_buffer_to_commit_point`.
- External deps: none beyond stdlib.
- Internal callers (best effort, slice 2 reconciles): scheduled to be wired into `src-tauri/src/commands/live.rs` (brief items #21, #24, #25). Comments note the integration ships in follow-up commits so threshold tuning can be validated against real microphone captures.
Public surface:
- `pub trait VadChunker: Send` — `crates/transcription/src/streaming/mod.rs:44`. Methods: `push`, `flush`, `reset`, `next_sample_index`.
- `pub struct VadChunk { start_sample: u64, samples: Vec<f32> }` — `crates/transcription/src/streaming/mod.rs:22`.
- `pub struct RmsVadChunker` — `crates/transcription/src/streaming/rms_vad.rs:57`.
- `pub fn RmsVadChunker::new() -> Self` — `crates/transcription/src/streaming/rms_vad.rs:90`.
- `pub fn RmsVadChunker::with_thresholds(...) -> Self` — `crates/transcription/src/streaming/rms_vad.rs:100`.
- `pub fn RmsVadChunker::sample_rate_hz(&self) -> u32` — `crates/transcription/src/streaming/rms_vad.rs:128`.
- `pub struct Token { text, start_secs, end_secs }` — `crates/transcription/src/streaming/commit_policy.rs:28`. PartialEq is text-only.
- `pub enum CommitPolicy { LocalAgreement { n: usize } }` — `crates/transcription/src/streaming/commit_policy.rs:57`. Default is `n = 2`.
- `pub struct CommitDecision { newly_committed: Vec<Token>, tentative: Vec<Token> }` — `crates/transcription/src/streaming/commit_policy.rs:44`.
- `pub struct LocalAgreement` — `crates/transcription/src/streaming/commit_policy.rs:77`.
- `pub fn LocalAgreement::{new, from_policy, push, flush, last_committed_end_secs, reset}` — same file.
- `pub fn sample_index_for_seconds(end_secs: f64, sample_rate: u32) -> u64` — `crates/transcription/src/streaming/buffer_trim.rs:23`.
- `pub fn trim_buffer_to_commit_point(buffer: &mut Vec<f32>, buffer_start_sample: u64, commit_sample_index: u64) -> u64` — `crates/transcription/src/streaming/buffer_trim.rs:40`.
## What's in here
### `VadChunker` trait (`streaming/mod.rs:44`)
`Send`-bound so a chunker can be moved into `spawn_blocking`. Object-safe (compile-time witness at `streaming/mod.rs:77`). Three methods plus an absolute-position accessor:
- `push(&mut self, samples: &[f32]) -> Vec<VadChunk>` — feed samples, get any chunks ready to dispatch.
- `flush(&mut self) -> Vec<VadChunk>` — end-of-session, surface any in-flight speech (note: returns `Vec`, not `Option`, because the zero-padded final frame can legitimately trigger both a mid-flush emission and a closing emission).
- `reset(&mut self)` — drop accumulated state.
- `next_sample_index(&self) -> u64` — what the next sample fed via `push` will be at, used by the commit-policy / buffer-trim glue.
### `RmsVadChunker` (`streaming/rms_vad.rs`)
Energy-backed VAD with hysteresis. Constants:
- `FRAME_SAMPLES = 800` (50 ms at 16 kHz).
- `DEFAULT_ENTER_RMS_THRESHOLD = 0.003`, `DEFAULT_EXIT_RMS_THRESHOLD = 0.0014` — separate thresholds give hysteresis so a score bouncing on a single value doesn't toggle state every frame.
- `DEFAULT_SPEECH_ONSET_FRAMES = 3` (3 sustained frames over enter threshold required to enter speech state — filters keyboard clicks).
- `DEFAULT_SILENCE_CLOSE_SAMPLES = 8_000` (500 ms of sub-threshold to close).
- `DEFAULT_MAX_CHUNK_SAMPLES = 32_000` (2 s, hard cap so Whisper isn't fed a 30-second monolith).
- `DEFAULT_SAMPLE_RATE_HZ = 16_000`.
Internal state machine (`streaming/rms_vad.rs:46`): `Idle` ↔ `InSpeech`. The `onset_buffer` keeps a rolling `speech_onset_frames * FRAME_SAMPLES` worth of audio so when speech is confirmed, the emitted chunk includes the onset itself, not just the post-onset audio.
Two emit paths:
- `emit_active_chunk_and_close` (`streaming/rms_vad.rs:217`) — end-of-utterance. Trims trailing silence, returns to `Idle`.
- `emit_active_chunk_continue` (`streaming/rms_vad.rs:248`) — hit `max_chunk_samples` mid-speech. Stays in `InSpeech`, advances `active_chunk_start` by the emitted length so the next chunk's start sample is contiguous.
`flush` is unusually careful (`streaming/rms_vad.rs:294`): it pads any sub-frame `pending` buffer to a full frame with zeros, runs `consume_frame` on it (which may emit a chunk), then if state is still `InSpeech` emits the active chunk as a closing chunk. Both emissions are returned. A 2026-04-22 audit (CRITICAL C2) found the prior code dropped the consume_frame chunk via `let _ = consume_frame(...)`; regression tests at `streaming/rms_vad.rs:567` and `:614` keep this honest.
### `LocalAgreement` (`streaming/commit_policy.rs`)
Source: ufal/whisper_streaming. Algorithm:
- Hold the last `n` ASR passes (default `n = 2`).
- The longest common prefix of those passes is the "agreed" prefix.
- Any tokens beyond that prefix are tentative (UI dashed-underline per workstream-B).
- Commit only grows: once a token is committed, even a divergent later pass cannot uncommit it (`new_committed = lcp_len.max(self.committed_count)`).
`Token` equality is **text-only** (`streaming/commit_policy.rs:34`); timestamps drift slightly between overlapping Whisper windows. Start/end seconds are absolute (session-relative) so the buffer-trim layer can compute sample indices.
`CommitDecision` has two vecs:
- `newly_committed` — append to the frontend's committed list.
- `tentative` — replaces (not appends to) the previous tentative list.
`flush` (`streaming/commit_policy.rs:163`) is end-of-stream: anything still tentative in the latest pass is committed and returned. Updates `last_committed_end_secs` so the buffer trim works on the closing region.
Defensive bookkeeping at `streaming/commit_policy.rs:131`: a later pass can legitimately arrive **shorter** than `committed_count` (Whisper re-transcribing an overlapping window with fewer segments). All slice indices are clamped against `latest.len()` to avoid panics. Regression test at `streaming/commit_policy.rs:317`.
### Buffer trim (`streaming/buffer_trim.rs`)
`sample_index_for_seconds(end_secs, sample_rate) -> u64`:
- Returns `0` for non-finite or `<= 0` inputs (defensive — without `is_finite`, `f64::INFINITY` would saturate to `u64::MAX` and trim the whole buffer forever).
- `(end_secs * sample_rate as f64).round() as u64` otherwise.
`trim_buffer_to_commit_point(buffer, buffer_start_sample, commit_sample_index) -> u64`:
- If the commit point is at or before the current buffer origin, do nothing.
- If it's past the buffer end, clear the buffer and park the new origin at the commit point.
- Otherwise drain the prefix and return the new origin.
The integrated property test at `streaming/buffer_trim.rs:144` simulates 100 cycles of 16 kHz captures and proves the buffer envelope stays bounded by `2 * tentative_per_cycle` instead of growing linearly.
## Data flow
```
StreamingResampler 16 kHz mono
└─ capture_buffer (Vec<f32>, anchored at buffer_start_sample)
└─ RmsVadChunker.push(samples) → Vec<VadChunk>
└─ for each VadChunk:
spawn_blocking → LocalEngine.transcribe_sync(chunk)
→ Vec<Segment>
→ tokens (Token { text, start_secs, end_secs })
→ LocalAgreement.push(tokens) → CommitDecision
├─ newly_committed → emit to frontend
│ └─ LocalAgreement.last_committed_end_secs
│ → sample_index_for_seconds
│ → trim_buffer_to_commit_point(capture_buffer)
└─ tentative → replace tentative tail in frontend
…session ends…
└─ RmsVadChunker.flush() → final Vec<VadChunk>
└─ LocalAgreement.flush() → final tentative committed
```
## Watch-outs
- **Not yet wired into `live.rs`.** All three primitives are unit-tested but their integration into `src-tauri/src/commands/live.rs` is explicitly described as a follow-up. Slice 2 owns the verification of whether that has landed.
- **Token equality is text-only.** Two passes that produce the same word with different casing or punctuation will not match. Whisper is fairly stable here, but a model swap is worth watching.
- **`speech_onset_frames` is a transient filter, not a denoiser.** Sustained background talk above the enter threshold will register as speech.
- **`FRAME_SAMPLES = 800` assumes 16 kHz.** A future backend at 24 kHz would need a different frame size to hit the same 50 ms window. `RmsVadChunker::sample_rate_hz` returns the constant; do not lie to it.
- **`reset()` zeros `next_sample_index`** but `flush()` deliberately does not (so a flush at end-of-session leaves the running counter accurate for any post-flush diagnostic).
- **`max_chunk_samples` is a hard 2 s cap.** A user reading aloud without a breath crosses it constantly. The continue-emit logic preserves audio contiguity (regression test at `streaming/rms_vad.rs:476`), but the resulting chunk boundaries land mid-word, which the `LocalAgreement` two-pass overlap can stitch back together — *if* the live session is feeding overlapping windows. If `live.rs` ever feeds non-overlapping chunks, expect words sliced in half.
- **`LocalAgreement::flush` may double-commit.** Calling it twice on the same instance: the second call would see `latest.len() == committed_count` and return empty. The `flush_with_no_history_is_empty` test pins this. Callers should still only call once at session end.
- **Buffer-trim's `sample_index_for_seconds` rounds nearest.** Integer comparisons against the trimmed buffer must not assume floor.
## See also
- [Audio capture pipeline](audio-capture-pipeline.md) — produces the chunks fed into `StreamingResampler`.
- [Audio resampling](audio-resampling.md) — `StreamingResampler` upstream of the chunker.
- [Transcription engines overview](transcription-engines-overview.md) — `LocalEngine.transcribe_sync` is what the streaming layer calls per chunk.
- [Audio VAD](audio-vad.md) — the deferred Silero path that will eventually replace `RmsVadChunker`.
- `docs/whisper-ecosystem/workstream-A.md`, `workstream-B.md` — the design source for thresholds and commit/tentative UX.
- Tests in `streaming/rms_vad.rs:359`, `streaming/commit_policy.rs:212`, `streaming/buffer_trim.rs:57`.

91
docs/architecture-map/03-audio-transcription/transcription-whisper.md Normal file
View File

@@ -0,0 +1,91 @@
---
name: Whisper backend (whisper-rs)
type: architecture-map-page
slice: 03-audio-transcription
last_verified: 2026/05/09
---
# Whisper backend (whisper-rs)
> **Where you are:** [Architecture map](../README.md) → [Audio + Transcription](README.md) → Whisper backend
**Plain English summary.** `WhisperRsBackend` owns a `WhisperContext` from `whisper-rs` 0.16 and runs Whisper directly (not via `transcribe-rs`). It is the only backend that pipes Magnotia's `initial_prompt` into the model. Each call to `transcribe_sync` builds a fresh `WhisperState`, applies language and prompt parameters, picks a thread count via the power-aware helper, and converts segment timestamps from centiseconds to seconds before returning `Vec<Segment>`. Vulkan GPU offload is additive behind a Cargo feature.
## At a glance
- Crate: `magnotia-transcription`
- Path: `crates/transcription/src/whisper_rs_backend.rs`
- LOC: 128
- Cargo feature gate: `whisper` (default on). Module is excluded entirely when the feature is off.
- External deps: `whisper-rs 0.16` (with optional `vulkan` sub-feature via `whisper-vulkan`), `thiserror 2`, `tracing 0.1`.
- Internal callers (best effort, slice 2 reconciles): `local_engine::load_whisper` (`crates/transcription/src/local_engine.rs:208`) constructs it; production load path runs through `src-tauri/src/commands/models.rs`.
Public surface:
- `pub struct WhisperRsBackend` — `crates/transcription/src/whisper_rs_backend.rs:30`. Field: `ctx: WhisperContext`.
- `pub fn WhisperRsBackend::load(model_path: &Path) -> std::result::Result<Self, WhisperBackendError>` — `crates/transcription/src/whisper_rs_backend.rs:35`.
- `pub enum WhisperBackendError { Load(String), State(String), Transcribe(String) }` — `crates/transcription/src/whisper_rs_backend.rs:21`. `thiserror`-derived; convertible to `MagnotiaError::TranscriptionFailed` via `to_string()`.
- `impl Transcriber for WhisperRsBackend` — `crates/transcription/src/whisper_rs_backend.rs:42`.
## What's in here
### `load`
`whisper_rs_backend.rs:35`. `WhisperContext::new_with_params(model_path, WhisperContextParameters::default())`. Errors map to `WhisperBackendError::Load`. The context is the heavy object (mmap'd GGML tensors, GPU buffers if Vulkan is active). It is held for the life of the backend.
### `capabilities`
`whisper_rs_backend.rs:43`. Returns `sample_rate = WHISPER_SAMPLE_RATE` (slice 5 constant), `channels = 1`, `supports_initial_prompt = true`. The truth flag is what tells the Settings UI to surface the prompt field.
### `transcribe_sync`
`whisper_rs_backend.rs:56`. The actual inference path:
1. `tracing::info!` boundary log capturing `language` and `has_initial_prompt` — used by the diagnostic bundle to confirm the prompt actually reached the backend.
2. `self.ctx.create_state()` — fresh `WhisperState` per call. The header notes "state can be reused, but fresh-per-call is simpler and matches the transcribe-rs call style we are replacing".
3. `FullParams::new(SamplingStrategy::Greedy { best_of: 1 })`.
4. Optional `params.set_language(Some(lang))` if `options.language` is set and non-empty.
5. Optional `params.set_initial_prompt(prompt)` if `options.initial_prompt` is set and non-empty.
6. `params.set_n_threads(inference_thread_count(Workload::Whisper, gpu_offloaded) as i32)` where `gpu_offloaded = cfg!(feature = "whisper-vulkan") && vulkan_loader_available()`. Power-aware (`magnotia_core::tuning::inference_thread_count` reads battery vs AC; helper returns a smaller count on battery to preserve runtime).
7. `set_print_special(false)`, `set_print_progress(false)`, `set_print_realtime(false)` — keep stdout silent.
8. `state.full(params, samples)` — runs inference.
9. Loops `state.full_n_segments()` calling `state.get_segment(i)`, pulling `seg.to_str()`, and converting timestamps: `start = seg.start_timestamp() as f64 * 0.01`, `end = seg.end_timestamp() as f64 * 0.01` (whisper-rs reports centiseconds).
10. Returns `Vec<Segment>`.
Errors all flow through `MagnotiaError::TranscriptionFailed(WhisperBackendError::*.to_string())`.
## Data flow
```
samples (f32, 16 kHz mono)
options: TranscriptionOptions { language, initial_prompt }
└─ tracing::info(boundary)
└─ WhisperState (fresh per call)
└─ FullParams (greedy, best_of=1)
├─ set_language(options.language)
├─ set_initial_prompt(options.initial_prompt)
├─ set_n_threads(inference_thread_count(Whisper, gpu_offloaded))
└─ silence stdout flags
└─ state.full(params, samples)
└─ for i in 0..state.full_n_segments():
Segment { start = t0 * 0.01, end = t1 * 0.01, text = seg.to_str() }
→ Vec<Segment>
```
## Watch-outs
- **`whisper-rs-sys` is the C++ link target.** Brief item #6 documents the historical Whispering v7.11.0 failure linking `whisper-rs-sys` + `tokenizers` together on Windows MSVC. The `build.rs` guard panics if `tokenizers` ever lands in the workspace lockfile on a Windows target. See [build-tokenizers-guard.md](build-tokenizers-guard.md).
- **Vulkan is detected at runtime, not just compile time.** `cfg!(feature = "whisper-vulkan")` is the static check; `vulkan_loader_available()` (slice 5 `magnotia_core::hardware`) confirms `libvulkan.so` actually resolves. If the feature is on but the loader is missing, the helper falls back to CPU thread count.
- **Fresh state per call has a cost.** `state.full` warm-up is lower than `WhisperContext` load but non-zero. Brief comment hints reuse is possible. Worth measuring once the live-streaming path is dogfooded.
- **No diarisation, no temperature fallback, no token suppression flags.** Greedy with `best_of: 1` is the simplest correct path. Tuning later is a single function.
- **Timestamp units.** whisper-rs returns centiseconds (10 ms). Multiplying by `0.01` gives seconds. A prior version that used milliseconds or token offsets would slice timing wrong; tests should pin this.
- **`initial_prompt` is the differentiator from `transcribe-rs`.** The whole point of writing this backend (vs going through `SpeechModelAdapter`) is that the adapter cannot pipe the prompt. Anyone tempted to "simplify" by routing Whisper through the adapter would silently drop the user-vocabulary feature.
- **Tracing field name typo immune.** `language = ?options.language` uses Debug formatting. If `Language` ever stops implementing `Debug`, this breaks at compile time, not at runtime.
## See also
- [Transcription engines overview](transcription-engines-overview.md) — the `Transcriber` trait this implements.
- [Cargo features](cargo-features.md) — `whisper` and `whisper-vulkan` flag matrix.
- [Build tokenizers guard](build-tokenizers-guard.md) — the Windows MSVC-CRT defence linked to this backend.
- [Tests and fixtures](tests-and-fixtures.md) — `whisper_rs_smoke.rs` exercises the load + transcribe + initial-prompt path; `jfk_bench.rs` measures cold/warm RTF; `thread_sweep.rs` validates thread-count scaling.
- `magnotia_core::hardware::vulkan_loader_available`, `magnotia_core::tuning::{inference_thread_count, Workload}` (slice 5).