jake/Lumotia
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
Lumotia/docs/architecture-map/03-audio-transcription/README.md
Jake 26c7307607
Some checks failed
check / cargo check (macos-latest) (push) Has been cancelled
Details
check / cargo check (ubuntu-22.04) (push) Has been cancelled
Details
check / cargo check (windows-latest) (push) Has been cancelled
Details
check / svelte build + lint (push) Has been cancelled
Details
agent: lumotia-rebrand — docs, scripts, root config, residuals 
Phase 9 of the rebrand cascade. Sweep covers everything the Phase 8
frontend pass deliberately skipped: docs/, root markdown, scripts,
Cargo.toml descriptions, code comments that survived earlier
word-boundary sed, plus a handful of identifiers caught on the final
verify pass.

transcription-app changes:
- README.md, HANDOVER.md, KNOWN-ISSUES.md, run.sh — magnotia/Magnotia
  -> lumotia/Lumotia.
- docs/ — sweep across all subdirs except docs/handovers/ (preserved
  as immutable audit trail). Includes architecture-map references
  to magnotia_core::*, magnotia_storage::*, etc. now pointing at
  lumotia_*; dev-setup.md tracing output examples (lumotia_startup
  target); brief/ + superpowers/ + issues/ + whisper-ecosystem/ +
  audit/.
- Cargo.toml descriptions on 9 crates (core, audio, cloud-providers,
  hotkey, llm, mcp, plus referenced others).
- crates/core/src/{error,hardware,recommendation,paths}.rs +
  crates/audio/src/wav.rs + crates/llm/src/model_manager.rs +
  crates/cloud-providers/src/keystore.rs + crates/mcp/src/lib.rs —
  doc comments and a model-manager user-agent string.
- Caught on final pass: BroadcastChannel("magnotia_task_sync") -> ...
  ("lumotia_task_sync"); magnotia_locale i18n localStorage key
  renamed + migration shim added; CSS keyframe names
  magnotiaPulse / magnotiaBar / magnotiaFade renamed in the design-
  system kit; magnotia_viewer_item / magnotia_viewer_mode handoff
  keys renamed in HistoryPage + viewer/+page.svelte; src/assets/
  wordmark.svg text.
- src-tauri/src/lib.rs comment cleanup ("magnotia era" was sed'd
  to "lumotia era" earlier — restored).

Preserved (intentional):
- crates/core/src/paths.rs — keeps "magnotia" / "Magnotia" / ".magnotia"
  legacy detection strings in legacy_and_target_paths() so the
  migration shim can still find user data from the magnotia era.
- src/lib/stores/{page,focusTimer}.svelte.ts + src/lib/i18n/index.ts
  — migration call sites reference the legacy magnotia keys
  deliberately.
- docs/handovers/ — historical audit trail.

cargo build --workspace passes. npm run check: 0 errors / 0 warnings
(3958 files). cargo test --workspace: 339 pass / 0 fail.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 12:38:03 +01:00

124 lines
9.8 KiB
Markdown
Raw Permalink Blame History

---
name: Slice 3 — Audio + Transcription
type: architecture-map-page
slice: 03-audio-transcription
last_verified: 2026/05/09
---
# Slice 3: Audio + Transcription
> **Where you are:** [Architecture map](../README.md) → Audio + Transcription
**Plain English summary.** Two Rust workspace crates form the backbone of Lumotia's speech-to-text path. `lumotia-audio` captures microphone input via `cpal`, decodes audio files via `symphonia`, resamples to 16 kHz mono via `rubato`, and writes WAV files via `hound`. `lumotia-transcription` loads Whisper or Parakeet models, runs inference, and provides streaming primitives (VAD chunking, LocalAgreement-n commit policy, commit-bounded buffer trim) so live captures stay responsive.
## At a glance
| Crate | LOC (`src/`) | Purpose |
|---|---|---|
| `lumotia-audio` | 1,533 | Capture, VAD stub, resample, decode, WAV I/O |
| `lumotia-transcription` | 2,266 (incl. tests + build) | Engines, model manager, streaming primitives |
Public crate surface (re-exports from `lib.rs`):
```rust
// lumotia-audio
pub use capture::{AudioChunk, CaptureRuntimeError, DeviceInfo, MicrophoneCapture};
pub use concurrency::decode_and_resample;
pub use decode::{decode_audio_file, decode_audio_file_limited, probe_audio_duration_secs};
pub use resample::resample_to_16khz;
pub use streaming_resample::StreamingResampler;
pub use vad::SpeechDetector;
pub use wav::{read_wav, write_wav, WavWriter};
// lumotia-transcription
pub use concurrency::run_inference;
#[cfg(feature = "whisper")]
pub use local_engine::load_whisper;
pub use local_engine::{load_parakeet, LocalEngine, SpeechModelAdapter, TimedTranscript};
pub use model_manager::{download, is_downloaded, list_downloaded, model_dir, models_dir};
pub use streaming::{
sample_index_for_seconds, trim_buffer_to_commit_point, CommitDecision, CommitPolicy,
LocalAgreement, RmsVadChunker, Token, VadChunk, VadChunker,
};
pub use transcribe_rs::SpeechModel;
pub use transcriber::{Transcriber, TranscriberCapabilities};
```
External deps that matter:
- `cpal 0.17` — microphone capture, host enumeration, stream callbacks.
- `rubato 0.15` — sinc-interpolating resampler (file + streaming).
- `symphonia 0.5` (mp3, aac, flac, pcm, vorbis, wav, ogg, isomp4) — file decode.
- `hound 3.5` — WAV reader and crash-safe append writer.
- `whisper-rs 0.16` (optional, default) — direct Whisper inference, pipes `initial_prompt`.
- `transcribe-rs 0.3` (onnx) — Parakeet wrapper.
- `reqwest 0.12` (rustls-tls, stream) + `sha2 0.10` + `futures-util 0.3` — model downloads with Range resume + SHA verify.
- `tokio 1` — `spawn_blocking` for inference and decode.
- `tracing 0.1` — backend boundary observability.
- `voice_activity_detector` / `silero-vad-rust` — **deferred** (ort 2.0.0-rc.10 vs 2.0.0-rc.12 conflict; see `audio-vad.md`).
Cargo feature matrix (`lumotia-transcription`):
| Feature | Default | Gates |
|---|---|---|
| `whisper` | yes | `whisper-rs` dep, `whisper_rs_backend` module, `load_whisper` fn |
| `whisper-vulkan` | yes | `whisper-rs/vulkan` (Vulkan GPU offload) |
| Parakeet (`transcribe-rs`) | always on | unconditional dep, no feature flag |
`lumotia-audio` has no Cargo features.
## Map of this slice
- [`audio-capture-pipeline.md`](audio-capture-pipeline.md) — `cpal` enumeration, monitor-source detection, RMS validation, hot-unplug error forwarding.
- [`audio-resampling.md`](audio-resampling.md) — `resample_to_16khz` (file) and `StreamingResampler` (live).
- [`audio-vad.md`](audio-vad.md) — `SpeechDetector` (current stub) and the ort version conflict that blocks Silero.
- [`audio-file-decoding.md`](audio-file-decoding.md) — `symphonia` decode, `decode_and_resample` async wrapper, `probe_audio_duration_secs`.
- [`audio-wav-io.md`](audio-wav-io.md) — `WavWriter` (crash-safe append) and `read_wav` / `write_wav` round-trip helpers.
- [`audio-pcm-bridge.md`](audio-pcm-bridge.md) — `static/pcm-processor.js` AudioWorklet (frontend-side fallback path).
- [`transcription-engines-overview.md`](transcription-engines-overview.md) — `Transcriber` trait, `TranscriberCapabilities`, `LocalEngine`.
- [`transcription-whisper.md`](transcription-whisper.md) — `WhisperRsBackend`, `WhisperContext`, params, `initial_prompt`, GPU offload.
- [`transcription-parakeet.md`](transcription-parakeet.md) — `SpeechModelAdapter` + `ParakeetWordGranularity`.
- [`transcription-streaming.md`](transcription-streaming.md) — `VadChunker` trait, `RmsVadChunker`, `LocalAgreement`, `trim_buffer_to_commit_point`.
- [`transcription-model-manager.md`](transcription-model-manager.md) — download flow, SHA verify, `.lumotia-verified` manifest, Range resume.
- [`transcription-concurrency.md`](transcription-concurrency.md) — `run_inference` async wrapper around `spawn_blocking`.
- [`cargo-features.md`](cargo-features.md) — feature matrix, build commands, rationale.
- [`build-tokenizers-guard.md`](build-tokenizers-guard.md) — `build.rs` Windows-MSVC-CRT guard against `tokenizers`.
- [`tests-and-fixtures.md`](tests-and-fixtures.md) — `thread_sweep.rs`, `whisper_rs_smoke.rs`, `jfk_bench.rs`, env-var gates, in-tree download server fixture.
## How this slice connects to others
- **Slice 2 (Tauri runtime)** owns `src-tauri/src/commands/{audio,transcription,live,models}.rs`. Those wrappers call into this slice via the `pub use` exports above. The live command in particular drives `MicrophoneCapture` + `StreamingResampler` + `RmsVadChunker` + `LocalAgreement` + `WavWriter` together. This crate publishes the primitives; slice 2 publishes the orchestrator.
- **Slice 4 (LLM + AI formatting)** runs after this slice produces a `Transcript`. It consumes the segment text via the storage layer, never directly. No type traffic flows back into this slice from formatting.
- **Slice 5 (core / storage / hotkey / build)** provides the shared types this slice depends on:
- `lumotia_core::error::{MagnotiaError, Result}` — error envelope.
- `lumotia_core::types::{AudioSamples, Segment, Transcript, TranscriptionOptions, EngineName, ModelId, DownloadProgress, Megabytes}`.
- `lumotia_core::constants::{WHISPER_SAMPLE_RATE, VAD_SPEECH_THRESHOLD}`.
- `lumotia_core::hardware::vulkan_loader_available` — runtime Vulkan probe used by `WhisperRsBackend`.
- `lumotia_core::tuning::{inference_thread_count, Workload}` — power-aware thread count picker.
- `lumotia_core::paths::app_paths` — `models_dir()` / `speech_model_dir()` resolution.
- `lumotia_core::model_registry::{find_model, all_models, ModelEntry, ModelFile}` — declarative model catalogue.
- Storage (slice 5) writes the produced `Transcript` to disk; this slice does not call into it directly.
## Open questions / debt
- **Silero VAD blocked on ort version conflict.** `crates/audio/src/vad.rs` is a stub that returns `true` for every input. Both `voice_activity_detector` and `silero-vad-rust` pin `ort 2.0.0-rc.10`; `transcribe-rs` (Parakeet) requires `2.0.0-rc.12`. The `RmsVadChunker` (transcription crate) is the live-capture fallback while this is unresolved.
- **VAD lives in two crates.** `audio::vad::SpeechDetector` is a single-frame predicate; `transcription::streaming::RmsVadChunker` is a stateful chunker. They share neither code nor thresholds. When Silero lands, expect consolidation.
- **Two resamplers maintained in parallel.** `resample::resample_to_16khz` (file) and `streaming_resample::StreamingResampler` (live) duplicate `SincInterpolationParameters` config. Drift between them will show up as audible mismatch between file imports and live captures.
- **Streaming primitives not yet wired into `live.rs`.** Comments in `streaming/mod.rs`, `buffer_trim.rs`, and `commit_policy.rs` flag that the integration into `src-tauri/src/commands/live.rs` ships as follow-up commits. Slice 2 is the place to verify whether that has landed.
- **`WhisperRsBackend` builds a fresh `WhisperState` per call.** Comment notes "state can be reused, but fresh-per-call is simpler". Worth measuring once we have the live-streaming path producing many small calls per session.
- **Parakeet quantisation is hardcoded to `Int8`** (`local_engine::load_parakeet`). No pathway for selecting a different quant.
- **Hardcoded `set_n_threads` in tests** (`jfk_bench.rs` uses 6) versus the production helper (`inference_thread_count(Workload::Whisper, gpu_offloaded)`). Test results are not comparable to runtime numbers without re-running with the helper-picked thread count.
- **`build.rs` panics at link time** on Windows if `tokenizers` ever enters the dep graph. This is intentional defence (Whispering v7.11.0 incident referenced in the file) but means a transitive pull of `tokenizers` from any sibling crate is a hard build break — not a soft warning. Worth surfacing in a top-level CONTRIBUTING note.
- **Symphonia feature list is fixed.** The `mp3, aac, flac, pcm, vorbis, wav, ogg, isomp4` set covers the import paths in v0.x. AAC requires patent-licensed code paths in some build configs — flagged by the brief.
## Existing in-repo docs (do not duplicate)
- `docs/whisper-ecosystem/brief.md` — top-level Whisper-ecosystem audit (the items #6, #8, #13, #19, #21, #24, #25, #26 referenced throughout this slice).
- `docs/whisper-ecosystem/workstream-A.md` — VAD / streaming roadmap (the source of `RmsVadChunker` thresholds).
- `docs/whisper-ecosystem/workstream-B.md` — UI commit/tentative contract that drives `LocalAgreement`.
- `docs/whisper-ecosystem/lumotia-context.md` — context summary for the workstreams.
- `docs/code-review-2026-04-22.md` — audit pass that flagged decode.rs RB-09 and `read_wav` filter_map bug (both fixed in tree).
- `docs/issues/decoder-partial-audio-on-error.md` — the RB-09 ticket.
- `docs/issues/native-capture-worker-join.md`, `c1-live-session-race.md`, `run-live-session-monolith.md` — slice 2's live-session debt; they reference primitives in this slice.
- `docs/brief/technology-map.md` — top-level tech map (covers all crates, not just this slice).
Reference in New Issue View Git Blame Copy Permalink