Lumotia/docs/architecture-map/03-audio-transcription

name, type, slice, last_verified

name	type	slice	last_verified
Slice 3 — Audio + Transcription	architecture-map-page	03-audio-transcription	2026/05/09

Slice 3: Audio + Transcription

Where you are: Architecture map → Audio + Transcription

Plain English summary. Two Rust workspace crates form the backbone of Lumotia's speech-to-text path. lumotia-audio captures microphone input via cpal, decodes audio files via symphonia, resamples to 16 kHz mono via rubato, and writes WAV files via hound. lumotia-transcription loads Whisper or Parakeet models, runs inference, and provides streaming primitives (VAD chunking, LocalAgreement-n commit policy, commit-bounded buffer trim) so live captures stay responsive.

At a glance

Crate	LOC (src/)	Purpose
lumotia-audio	1,533	Capture, VAD stub, resample, decode, WAV I/O
lumotia-transcription	2,266 (incl. tests + build)	Engines, model manager, streaming primitives

Public crate surface (re-exports from lib.rs):

// lumotia-audio
pub use capture::{AudioChunk, CaptureRuntimeError, DeviceInfo, MicrophoneCapture};
pub use concurrency::decode_and_resample;
pub use decode::{decode_audio_file, decode_audio_file_limited, probe_audio_duration_secs};
pub use resample::resample_to_16khz;
pub use streaming_resample::StreamingResampler;
pub use vad::SpeechDetector;
pub use wav::{read_wav, write_wav, WavWriter};

// lumotia-transcription
pub use concurrency::run_inference;
#[cfg(feature = "whisper")]
pub use local_engine::load_whisper;
pub use local_engine::{load_parakeet, LocalEngine, SpeechModelAdapter, TimedTranscript};
pub use model_manager::{download, is_downloaded, list_downloaded, model_dir, models_dir};
pub use streaming::{
    sample_index_for_seconds, trim_buffer_to_commit_point, CommitDecision, CommitPolicy,
    LocalAgreement, RmsVadChunker, Token, VadChunk, VadChunker,
};
pub use transcribe_rs::SpeechModel;
pub use transcriber::{Transcriber, TranscriberCapabilities};

External deps that matter:

• cpal 0.17 — microphone capture, host enumeration, stream callbacks.
• rubato 0.15 — sinc-interpolating resampler (file + streaming).
• symphonia 0.5 (mp3, aac, flac, pcm, vorbis, wav, ogg, isomp4) — file decode.
• hound 3.5 — WAV reader and crash-safe append writer.
• whisper-rs 0.16 (optional, default) — direct Whisper inference, pipes initial_prompt.
• transcribe-rs 0.3 (onnx) — Parakeet wrapper.
• reqwest 0.12 (rustls-tls, stream) + sha2 0.10 + futures-util 0.3 — model downloads with Range resume + SHA verify.
• tokio 1 — spawn_blocking for inference and decode.
• tracing 0.1 — backend boundary observability.
• voice_activity_detector / silero-vad-rust — deferred (ort 2.0.0-rc.10 vs 2.0.0-rc.12 conflict; see audio-vad.md).

Cargo feature matrix (lumotia-transcription):

Feature	Default	Gates
whisper	yes	whisper-rs dep, whisper_rs_backend module, load_whisper fn
whisper-vulkan	yes	whisper-rs/vulkan (Vulkan GPU offload)
Parakeet (transcribe-rs)	always on	unconditional dep, no feature flag

lumotia-audio has no Cargo features.

Map of this slice

• audio-capture-pipeline.md — cpal enumeration, monitor-source detection, RMS validation, hot-unplug error forwarding.
• audio-resampling.md — resample_to_16khz (file) and StreamingResampler (live).
• audio-vad.md — SpeechDetector (current stub) and the ort version conflict that blocks Silero.
• audio-file-decoding.md — symphonia decode, decode_and_resample async wrapper, probe_audio_duration_secs.
• audio-wav-io.md — WavWriter (crash-safe append) and read_wav / write_wav round-trip helpers.
• audio-pcm-bridge.md — static/pcm-processor.js AudioWorklet (frontend-side fallback path).
• transcription-engines-overview.md — Transcriber trait, TranscriberCapabilities, LocalEngine.
• transcription-whisper.md — WhisperRsBackend, WhisperContext, params, initial_prompt, GPU offload.
• transcription-parakeet.md — SpeechModelAdapter + ParakeetWordGranularity.
• transcription-streaming.md — VadChunker trait, RmsVadChunker, LocalAgreement, trim_buffer_to_commit_point.
• transcription-model-manager.md — download flow, SHA verify, .lumotia-verified manifest, Range resume.
• transcription-concurrency.md — run_inference async wrapper around spawn_blocking.
• cargo-features.md — feature matrix, build commands, rationale.
• build-tokenizers-guard.md — build.rs Windows-MSVC-CRT guard against tokenizers.
• tests-and-fixtures.md — thread_sweep.rs, whisper_rs_smoke.rs, jfk_bench.rs, env-var gates, in-tree download server fixture.

How this slice connects to others

• Slice 2 (Tauri runtime) owns src-tauri/src/commands/{audio,transcription,live,models}.rs. Those wrappers call into this slice via the pub use exports above. The live command in particular drives MicrophoneCapture + StreamingResampler + RmsVadChunker + LocalAgreement + WavWriter together. This crate publishes the primitives; slice 2 publishes the orchestrator.
• Slice 4 (LLM + AI formatting) runs after this slice produces a Transcript. It consumes the segment text via the storage layer, never directly. No type traffic flows back into this slice from formatting.
• Slice 5 (core / storage / hotkey / build) provides the shared types this slice depends on:
  • lumotia_core::error::{MagnotiaError, Result} — error envelope.
  • lumotia_core::types::{AudioSamples, Segment, Transcript, TranscriptionOptions, EngineName, ModelId, DownloadProgress, Megabytes}.
  • lumotia_core::constants::{WHISPER_SAMPLE_RATE, VAD_SPEECH_THRESHOLD}.
  • lumotia_core::hardware::vulkan_loader_available — runtime Vulkan probe used by WhisperRsBackend.
  • lumotia_core::tuning::{inference_thread_count, Workload} — power-aware thread count picker.
  • lumotia_core::paths::app_paths — models_dir() / speech_model_dir() resolution.
  • lumotia_core::model_registry::{find_model, all_models, ModelEntry, ModelFile} — declarative model catalogue.
  • Storage (slice 5) writes the produced Transcript to disk; this slice does not call into it directly.

Open questions / debt

• Silero VAD blocked on ort version conflict. crates/audio/src/vad.rs is a stub that returns true for every input. Both voice_activity_detector and silero-vad-rust pin ort 2.0.0-rc.10; transcribe-rs (Parakeet) requires 2.0.0-rc.12. The RmsVadChunker (transcription crate) is the live-capture fallback while this is unresolved.
• VAD lives in two crates. audio::vad::SpeechDetector is a single-frame predicate; transcription::streaming::RmsVadChunker is a stateful chunker. They share neither code nor thresholds. When Silero lands, expect consolidation.
• Two resamplers maintained in parallel. resample::resample_to_16khz (file) and streaming_resample::StreamingResampler (live) duplicate SincInterpolationParameters config. Drift between them will show up as audible mismatch between file imports and live captures.
• Streaming primitives not yet wired into live.rs. Comments in streaming/mod.rs, buffer_trim.rs, and commit_policy.rs flag that the integration into src-tauri/src/commands/live.rs ships as follow-up commits. Slice 2 is the place to verify whether that has landed.
• WhisperRsBackend builds a fresh WhisperState per call. Comment notes "state can be reused, but fresh-per-call is simpler". Worth measuring once we have the live-streaming path producing many small calls per session.
• Parakeet quantisation is hardcoded to Int8 (local_engine::load_parakeet). No pathway for selecting a different quant.
• Hardcoded set_n_threads in tests (jfk_bench.rs uses 6) versus the production helper (inference_thread_count(Workload::Whisper, gpu_offloaded)). Test results are not comparable to runtime numbers without re-running with the helper-picked thread count.
• build.rs panics at link time on Windows if tokenizers ever enters the dep graph. This is intentional defence (Whispering v7.11.0 incident referenced in the file) but means a transitive pull of tokenizers from any sibling crate is a hard build break — not a soft warning. Worth surfacing in a top-level CONTRIBUTING note.
• Symphonia feature list is fixed. The mp3, aac, flac, pcm, vorbis, wav, ogg, isomp4 set covers the import paths in v0.x. AAC requires patent-licensed code paths in some build configs — flagged by the brief.

Existing in-repo docs (do not duplicate)

• docs/whisper-ecosystem/brief.md — top-level Whisper-ecosystem audit (the items #6, #8, #13, #19, #21, #24, #25, #26 referenced throughout this slice).
• docs/whisper-ecosystem/workstream-A.md — VAD / streaming roadmap (the source of RmsVadChunker thresholds).
• docs/whisper-ecosystem/workstream-B.md — UI commit/tentative contract that drives LocalAgreement.
• docs/whisper-ecosystem/lumotia-context.md — context summary for the workstreams.
• docs/code-review-2026-04-22.md — audit pass that flagged decode.rs RB-09 and read_wav filter_map bug (both fixed in tree).
• docs/issues/decoder-partial-audio-on-error.md — the RB-09 ticket.
• docs/issues/native-capture-worker-join.md, c1-live-session-race.md, run-live-session-monolith.md — slice 2's live-session debt; they reference primitives in this slice.
• docs/brief/technology-map.md — top-level tech map (covers all crates, not just this slice).