docs: architecture map (initial 5-slice generation, 105 pages)
Five-slice navigable map of the entire codebase under
docs/architecture-map/. Each slice is a self-contained
breadcrumbed sub-tree:
01-frontend (16) Svelte/SvelteKit UI
02-tauri-runtime (26) src-tauri commands + lifecycle
03-audio-transcription (16) audio + transcription crates
04-llm-formatting-mcp (19) llm, ai-formatting, mcp, cloud
05-core-storage-hotkey-build core, storage, hotkey, workspace,
(26) CI, dev glue
Plus master README.md and data-flow-end-to-end.md tracing
audio bytes from microphone to FTS5 search to MCP read.
Generated by 5 parallel subagents on 2026/05/09 against
HEAD 3c47000. Each page has YAML frontmatter, file:line code
refs, sibling cross-links, plain-English summaries.
Aggregated debt surfaced (full lists in master README):
RB-08 macOS power assertion, schema head drift v14 vs v15,
VAD blocked on ort version conflict, streaming primitives
not wired into live.rs, no prompt versioning, MCP has no
auth, cloud-providers in-memory keystore, SettingsPage
2 484 LOC, commands/live.rs 1 737 LOC, dual theme system,
brand rename to Lumenote pending across the codebase.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jars
106
docs/architecture-map/03-audio-transcription/audio-resampling.md
Normal file
106
docs/architecture-map/03-audio-transcription/audio-resampling.md
Normal file
@@ -0,0 +1,106 @@
|
||||
---
|
||||
name: Audio resampling (rubato)
|
||||
type: architecture-map-page
|
||||
slice: 03-audio-transcription
|
||||
last_verified: 2026/05/09
|
||||
---
|
||||
|
||||
# Audio resampling (rubato)
|
||||
|
||||
> **Where you are:** [Architecture map](../README.md) → [Audio + Transcription](README.md) → Audio resampling
|
||||
|
||||
**Plain English summary.** Whisper and Parakeet want 16 kHz mono `f32`. Microphones and audio files almost never deliver that natively. Two `rubato`-backed resamplers convert: `resample_to_16khz` for one-shot file conversion, `StreamingResampler` for the live session where samples arrive in arbitrary chunks and a flush at the end drains the resampler's tail.
|
||||
|
||||
## At a glance
|
||||
|
||||
- Crate: `magnotia-audio`
|
||||
- Paths: `crates/audio/src/resample.rs` (100 LOC), `crates/audio/src/streaming_resample.rs` (211 LOC).
|
||||
- External deps: `rubato 0.15` (sinc interpolation).
|
||||
- Internal callers (best effort, slice 2 reconciles): `concurrency::decode_and_resample` (file path used by `audio.rs` and `transcription.rs` Tauri commands), `live.rs` Tauri command (live path).
|
||||
|
||||
Public surface:
|
||||
|
||||
- `pub fn resample_to_16khz(audio: &AudioSamples) -> Result<AudioSamples>` — `crates/audio/src/resample.rs:11`.
|
||||
- `pub enum StreamingResampler` — `crates/audio/src/streaming_resample.rs:37`. Variants: `Passthrough`, `Sinc { resampler, residual, ratio }`.
|
||||
- `pub fn StreamingResampler::new(from_rate: u32) -> Result<Self>` — `crates/audio/src/streaming_resample.rs:52`.
|
||||
- `pub fn StreamingResampler::push_samples(&mut self, mono: &[f32]) -> Result<Vec<f32>>` — `crates/audio/src/streaming_resample.rs:93`.
|
||||
- `pub fn StreamingResampler::flush(&mut self) -> Result<Vec<f32>>` — `crates/audio/src/streaming_resample.rs:127`.
|
||||
|
||||
## What's in here
|
||||
|
||||
### Shared sinc parameters
|
||||
|
||||
Both files use the same `SincInterpolationParameters`:
|
||||
|
||||
```rust
|
||||
sinc_len: 256,
|
||||
f_cutoff: 0.95,
|
||||
oversampling_factor: 128,
|
||||
interpolation: SincInterpolationType::Cubic,
|
||||
window: WindowFunction::Blackman,
|
||||
```
|
||||
|
||||
`max_relative_jitter = 1.1`, `chunks_in = 1024`, channels = 1.
|
||||
|
||||
### `resample_to_16khz` (file)
|
||||
|
||||
`crates/audio/src/resample.rs:11`. Pure function over an immutable `AudioSamples`. Short-circuits the passthrough case (already 16 kHz). Iterates the input in 1024-sample chunks; pads the trailing partial chunk to `chunk_size` zeros; truncates the output to `(input_len * ratio) as usize` to discard the padding's residue.
|
||||
|
||||
Errors map to `MagnotiaError::AudioDecodeFailed` (one variant for ratio init failure, one for per-chunk process failure).
|
||||
|
||||
### `StreamingResampler` (live)
|
||||
|
||||
`crates/audio/src/streaming_resample.rs`. Two states:
|
||||
|
||||
- `Passthrough` — emitted when `from_rate == WHISPER_SAMPLE_RATE`. `push_samples` returns input verbatim; `flush` returns empty.
|
||||
- `Sinc { resampler, residual, ratio }` — wraps a `SincFixedIn::<f32>`. `residual` buffers samples that don't yet form a complete `INPUT_CHUNK = 1024`. Each `push_samples` call drains as many full chunks as it can.
|
||||
|
||||
`flush()` is the load-bearing tail handler:
|
||||
|
||||
1. If `residual` is empty, return empty (no flush work pending).
|
||||
2. Pad `residual` up to `INPUT_CHUNK` zeros.
|
||||
3. Run `resampler.process()` once over the padded chunk.
|
||||
4. Truncate the output to `(leftover_real_samples * ratio).round() as usize` so the silence introduced by padding does not leak into the saved WAV.
|
||||
|
||||
The `flush` correctness math is verified by the `streaming_48k_to_16k_preserves_duration` test (`streaming_resample.rs:183`).
|
||||
|
||||
## Data flow
|
||||
|
||||
File path:
|
||||
|
||||
```
|
||||
AudioSamples (any rate, mono)
|
||||
└─ resample_to_16khz()
|
||||
├─ if rate == 16k → AudioSamples::mono_16khz(clone)
|
||||
└─ else: rubato SincFixedIn loop, padded final chunk, truncate to expected_len
|
||||
→ AudioSamples (16k, mono)
|
||||
```
|
||||
|
||||
Live path:
|
||||
|
||||
```
|
||||
cpal AudioChunk (native rate, possibly stereo, downmixed by caller)
|
||||
└─ StreamingResampler::push_samples(&mono_f32)
|
||||
├─ residual.extend(mono)
|
||||
├─ while residual.len() >= 1024: resampler.process(chunk) → out
|
||||
└─ return out (zero or more 16 kHz samples)
|
||||
...session ends...
|
||||
└─ StreamingResampler::flush()
|
||||
└─ pad residual to 1024, process, trim padding-induced output
|
||||
→ 16 kHz mono Vec<f32>
|
||||
```
|
||||
|
||||
## Watch-outs
|
||||
|
||||
- **Two implementations, one parameter set.** Drift between `resample.rs` and `streaming_resample.rs` is currently held by convention only. Any change to `SincInterpolationParameters` must touch both files.
|
||||
- **Caller is responsible for downmixing.** Both functions assume mono input. `decode.rs` averages channels for files; the live session must do the same before pushing to `StreamingResampler`.
|
||||
- **`from_rate = 0` is rejected** in both functions (matches a corrupt WAV header path).
|
||||
- **Truncation policy differs slightly.** `resample_to_16khz` uses `(samples.len() as f64 * ratio) as usize` (truncating cast), `StreamingResampler::flush` uses `.round() as usize`. Off by one sample at most; matters for WAV file length parity if you ever compare them.
|
||||
- **No streaming for file path.** `resample_to_16khz` materialises the whole resampled vec in memory. A 90-minute meeting at 48 kHz pre-resample is ~250 MB of `f32`; the post-resample buffer is another 350 MB. Acceptable today, brittle for very long imports.
|
||||
- **Passthrough still calls `to_vec()`** on the input samples (`streaming_resample.rs:95`). Cheap but not free; if push frequency is high consider returning `Cow`.
|
||||
|
||||
## See also
|
||||
|
||||
- [Audio capture pipeline](audio-capture-pipeline.md) — produces the `AudioChunk`s that feed `StreamingResampler`.
|
||||
- [Audio file decoding](audio-file-decoding.md) — `decode_and_resample` chains decode + `resample_to_16khz`.
|
||||
- `magnotia_core::constants::WHISPER_SAMPLE_RATE` (slice 5) — single source of truth for the target rate.
|
||||
Reference in New Issue
Block a user