jake/Lumotia
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: architecture map (initial 5-slice generation, 105 pages)

Browse Source 
Five-slice navigable map of the entire codebase under
docs/architecture-map/. Each slice is a self-contained
breadcrumbed sub-tree:

  01-frontend (16)              Svelte/SvelteKit UI
  02-tauri-runtime (26)         src-tauri commands + lifecycle
  03-audio-transcription (16)   audio + transcription crates
  04-llm-formatting-mcp (19)    llm, ai-formatting, mcp, cloud
  05-core-storage-hotkey-build  core, storage, hotkey, workspace,
                          (26) CI, dev glue

Plus master README.md and data-flow-end-to-end.md tracing
audio bytes from microphone to FTS5 search to MCP read.

Generated by 5 parallel subagents on 2026/05/09 against
HEAD 3c47000. Each page has YAML frontmatter, file:line code
refs, sibling cross-links, plain-English summaries.

Aggregated debt surfaced (full lists in master README):
RB-08 macOS power assertion, schema head drift v14 vs v15,
VAD blocked on ort version conflict, streaming primitives
not wired into live.rs, no prompt versioning, MCP has no
auth, cloud-providers in-memory keystore, SettingsPage
2 484 LOC, commands/live.rs 1 737 LOC, dual theme system,
brand rename to Lumenote pending across the codebase.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jars
2026-05-09 14:04:13 +01:00
parent 3c47000ea9
commit a1f3f3f134
105 changed files with 11266 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

123
docs/architecture-map/03-audio-transcription/README.md Normal file
View File

@@ -0,0 +1,123 @@
---
name: Slice 3 — Audio + Transcription
type: architecture-map-page
slice: 03-audio-transcription
last_verified: 2026/05/09
---
# Slice 3: Audio + Transcription
> **Where you are:** [Architecture map](../README.md) → Audio + Transcription
**Plain English summary.** Two Rust workspace crates form the backbone of Magnotia's speech-to-text path. `magnotia-audio` captures microphone input via `cpal`, decodes audio files via `symphonia`, resamples to 16 kHz mono via `rubato`, and writes WAV files via `hound`. `magnotia-transcription` loads Whisper or Parakeet models, runs inference, and provides streaming primitives (VAD chunking, LocalAgreement-n commit policy, commit-bounded buffer trim) so live captures stay responsive.
## At a glance
| Crate | LOC (`src/`) | Purpose |
|---|---|---|
| `magnotia-audio` | 1,533 | Capture, VAD stub, resample, decode, WAV I/O |
| `magnotia-transcription` | 2,266 (incl. tests + build) | Engines, model manager, streaming primitives |
Public crate surface (re-exports from `lib.rs`):
```rust
// magnotia-audio
pub use capture::{AudioChunk, CaptureRuntimeError, DeviceInfo, MicrophoneCapture};
pub use concurrency::decode_and_resample;
pub use decode::{decode_audio_file, decode_audio_file_limited, probe_audio_duration_secs};
pub use resample::resample_to_16khz;
pub use streaming_resample::StreamingResampler;
pub use vad::SpeechDetector;
pub use wav::{read_wav, write_wav, WavWriter};
// magnotia-transcription
pub use concurrency::run_inference;
#[cfg(feature = "whisper")]
pub use local_engine::load_whisper;
pub use local_engine::{load_parakeet, LocalEngine, SpeechModelAdapter, TimedTranscript};
pub use model_manager::{download, is_downloaded, list_downloaded, model_dir, models_dir};
pub use streaming::{
sample_index_for_seconds, trim_buffer_to_commit_point, CommitDecision, CommitPolicy,
LocalAgreement, RmsVadChunker, Token, VadChunk, VadChunker,
};
pub use transcribe_rs::SpeechModel;
pub use transcriber::{Transcriber, TranscriberCapabilities};
```
External deps that matter:
- `cpal 0.17` — microphone capture, host enumeration, stream callbacks.
- `rubato 0.15` — sinc-interpolating resampler (file + streaming).
- `symphonia 0.5` (mp3, aac, flac, pcm, vorbis, wav, ogg, isomp4) — file decode.
- `hound 3.5` — WAV reader and crash-safe append writer.
- `whisper-rs 0.16` (optional, default) — direct Whisper inference, pipes `initial_prompt`.
- `transcribe-rs 0.3` (onnx) — Parakeet wrapper.
- `reqwest 0.12` (rustls-tls, stream) + `sha2 0.10` + `futures-util 0.3` — model downloads with Range resume + SHA verify.
- `tokio 1` — `spawn_blocking` for inference and decode.
- `tracing 0.1` — backend boundary observability.
- `voice_activity_detector` / `silero-vad-rust` — **deferred** (ort 2.0.0-rc.10 vs 2.0.0-rc.12 conflict; see `audio-vad.md`).
Cargo feature matrix (`magnotia-transcription`):
| Feature | Default | Gates |
|---|---|---|
| `whisper` | yes | `whisper-rs` dep, `whisper_rs_backend` module, `load_whisper` fn |
| `whisper-vulkan` | yes | `whisper-rs/vulkan` (Vulkan GPU offload) |
| Parakeet (`transcribe-rs`) | always on | unconditional dep, no feature flag |
`magnotia-audio` has no Cargo features.
## Map of this slice
- [`audio-capture-pipeline.md`](audio-capture-pipeline.md) — `cpal` enumeration, monitor-source detection, RMS validation, hot-unplug error forwarding.
- [`audio-resampling.md`](audio-resampling.md) — `resample_to_16khz` (file) and `StreamingResampler` (live).
- [`audio-vad.md`](audio-vad.md) — `SpeechDetector` (current stub) and the ort version conflict that blocks Silero.
- [`audio-file-decoding.md`](audio-file-decoding.md) — `symphonia` decode, `decode_and_resample` async wrapper, `probe_audio_duration_secs`.
- [`audio-wav-io.md`](audio-wav-io.md) — `WavWriter` (crash-safe append) and `read_wav` / `write_wav` round-trip helpers.
- [`audio-pcm-bridge.md`](audio-pcm-bridge.md) — `static/pcm-processor.js` AudioWorklet (frontend-side fallback path).
- [`transcription-engines-overview.md`](transcription-engines-overview.md) — `Transcriber` trait, `TranscriberCapabilities`, `LocalEngine`.
- [`transcription-whisper.md`](transcription-whisper.md) — `WhisperRsBackend`, `WhisperContext`, params, `initial_prompt`, GPU offload.
- [`transcription-parakeet.md`](transcription-parakeet.md) — `SpeechModelAdapter` + `ParakeetWordGranularity`.
- [`transcription-streaming.md`](transcription-streaming.md) — `VadChunker` trait, `RmsVadChunker`, `LocalAgreement`, `trim_buffer_to_commit_point`.
- [`transcription-model-manager.md`](transcription-model-manager.md) — download flow, SHA verify, `.magnotia-verified` manifest, Range resume.
- [`transcription-concurrency.md`](transcription-concurrency.md) — `run_inference` async wrapper around `spawn_blocking`.
- [`cargo-features.md`](cargo-features.md) — feature matrix, build commands, rationale.
- [`build-tokenizers-guard.md`](build-tokenizers-guard.md) — `build.rs` Windows-MSVC-CRT guard against `tokenizers`.
- [`tests-and-fixtures.md`](tests-and-fixtures.md) — `thread_sweep.rs`, `whisper_rs_smoke.rs`, `jfk_bench.rs`, env-var gates, in-tree download server fixture.
## How this slice connects to others
- **Slice 2 (Tauri runtime)** owns `src-tauri/src/commands/{audio,transcription,live,models}.rs`. Those wrappers call into this slice via the `pub use` exports above. The live command in particular drives `MicrophoneCapture` + `StreamingResampler` + `RmsVadChunker` + `LocalAgreement` + `WavWriter` together. This crate publishes the primitives; slice 2 publishes the orchestrator.
- **Slice 4 (LLM + AI formatting)** runs after this slice produces a `Transcript`. It consumes the segment text via the storage layer, never directly. No type traffic flows back into this slice from formatting.
- **Slice 5 (core / storage / hotkey / build)** provides the shared types this slice depends on:
- `magnotia_core::error::{MagnotiaError, Result}` — error envelope.
- `magnotia_core::types::{AudioSamples, Segment, Transcript, TranscriptionOptions, EngineName, ModelId, DownloadProgress, Megabytes}`.
- `magnotia_core::constants::{WHISPER_SAMPLE_RATE, VAD_SPEECH_THRESHOLD}`.
- `magnotia_core::hardware::vulkan_loader_available` — runtime Vulkan probe used by `WhisperRsBackend`.
- `magnotia_core::tuning::{inference_thread_count, Workload}` — power-aware thread count picker.
- `magnotia_core::paths::app_paths` — `models_dir()` / `speech_model_dir()` resolution.
- `magnotia_core::model_registry::{find_model, all_models, ModelEntry, ModelFile}` — declarative model catalogue.
- Storage (slice 5) writes the produced `Transcript` to disk; this slice does not call into it directly.
## Open questions / debt
- **Silero VAD blocked on ort version conflict.** `crates/audio/src/vad.rs` is a stub that returns `true` for every input. Both `voice_activity_detector` and `silero-vad-rust` pin `ort 2.0.0-rc.10`; `transcribe-rs` (Parakeet) requires `2.0.0-rc.12`. The `RmsVadChunker` (transcription crate) is the live-capture fallback while this is unresolved.
- **VAD lives in two crates.** `audio::vad::SpeechDetector` is a single-frame predicate; `transcription::streaming::RmsVadChunker` is a stateful chunker. They share neither code nor thresholds. When Silero lands, expect consolidation.
- **Two resamplers maintained in parallel.** `resample::resample_to_16khz` (file) and `streaming_resample::StreamingResampler` (live) duplicate `SincInterpolationParameters` config. Drift between them will show up as audible mismatch between file imports and live captures.
- **Streaming primitives not yet wired into `live.rs`.** Comments in `streaming/mod.rs`, `buffer_trim.rs`, and `commit_policy.rs` flag that the integration into `src-tauri/src/commands/live.rs` ships as follow-up commits. Slice 2 is the place to verify whether that has landed.
- **`WhisperRsBackend` builds a fresh `WhisperState` per call.** Comment notes "state can be reused, but fresh-per-call is simpler". Worth measuring once we have the live-streaming path producing many small calls per session.
- **Parakeet quantisation is hardcoded to `Int8`** (`local_engine::load_parakeet`). No pathway for selecting a different quant.
- **Hardcoded `set_n_threads` in tests** (`jfk_bench.rs` uses 6) versus the production helper (`inference_thread_count(Workload::Whisper, gpu_offloaded)`). Test results are not comparable to runtime numbers without re-running with the helper-picked thread count.
- **`build.rs` panics at link time** on Windows if `tokenizers` ever enters the dep graph. This is intentional defence (Whispering v7.11.0 incident referenced in the file) but means a transitive pull of `tokenizers` from any sibling crate is a hard build break — not a soft warning. Worth surfacing in a top-level CONTRIBUTING note.
- **Symphonia feature list is fixed.** The `mp3, aac, flac, pcm, vorbis, wav, ogg, isomp4` set covers the import paths in v0.x. AAC requires patent-licensed code paths in some build configs — flagged by the brief.
## Existing in-repo docs (do not duplicate)
- `docs/whisper-ecosystem/brief.md` — top-level Whisper-ecosystem audit (the items #6, #8, #13, #19, #21, #24, #25, #26 referenced throughout this slice).
- `docs/whisper-ecosystem/workstream-A.md` — VAD / streaming roadmap (the source of `RmsVadChunker` thresholds).
- `docs/whisper-ecosystem/workstream-B.md` — UI commit/tentative contract that drives `LocalAgreement`.
- `docs/whisper-ecosystem/magnotia-context.md` — context summary for the workstreams.
- `docs/code-review-2026-04-22.md` — audit pass that flagged decode.rs RB-09 and `read_wav` filter_map bug (both fixed in tree).
- `docs/issues/decoder-partial-audio-on-error.md` — the RB-09 ticket.
- `docs/issues/native-capture-worker-join.md`, `c1-live-session-race.md`, `run-live-session-monolith.md` — slice 2's live-session debt; they reference primitives in this slice.
- `docs/brief/technology-map.md` — top-level tech map (covers all crates, not just this slice).