---
name: Slice 3 — Audio + Transcription
type: architecture-map-page
slice: 03-audio-transcription
last_verified: 2026/05/09
---

# Slice 3: Audio + Transcription

> **Where you are:** [Architecture map](../README.md) → Audio + Transcription

**Plain English summary.** Two Rust workspace crates form the backbone of Lumotia's speech-to-text path. `lumotia-audio` captures microphone input via `cpal`, decodes audio files via `symphonia`, resamples to 16 kHz mono via `rubato`, and writes WAV files via `hound`. `lumotia-transcription` loads Whisper or Parakeet models, runs inference, and provides streaming primitives (VAD chunking, LocalAgreement-n commit policy, commit-bounded buffer trim) so live captures stay responsive.

## At a glance

| Crate | LOC (`src/`) | Purpose |
|---|---|---|
| `lumotia-audio` | 1,533 | Capture, VAD stub, resample, decode, WAV I/O |
| `lumotia-transcription` | 2,266 (incl. tests + build) | Engines, model manager, streaming primitives |

Public crate surface (re-exports from `lib.rs`):

```rust
// lumotia-audio
pub use capture::{AudioChunk, CaptureRuntimeError, DeviceInfo, MicrophoneCapture};
pub use concurrency::decode_and_resample;
pub use decode::{decode_audio_file, decode_audio_file_limited, probe_audio_duration_secs};
pub use resample::resample_to_16khz;
pub use streaming_resample::StreamingResampler;
pub use vad::SpeechDetector;
pub use wav::{read_wav, write_wav, WavWriter};

// lumotia-transcription
pub use concurrency::run_inference;
#[cfg(feature = "whisper")]
pub use local_engine::load_whisper;
pub use local_engine::{load_parakeet, LocalEngine, SpeechModelAdapter, TimedTranscript};
pub use model_manager::{download, is_downloaded, list_downloaded, model_dir, models_dir};
pub use streaming::{
    sample_index_for_seconds, trim_buffer_to_commit_point, CommitDecision, CommitPolicy,
    LocalAgreement, RmsVadChunker, Token, VadChunk, VadChunker,
};
pub use transcribe_rs::SpeechModel;
pub use transcriber::{Transcriber, TranscriberCapabilities};
```

External deps that matter:

- `cpal 0.17` — microphone capture, host enumeration, stream callbacks.
- `rubato 0.15` — sinc-interpolating resampler (file + streaming).
- `symphonia 0.5` (mp3, aac, flac, pcm, vorbis, wav, ogg, isomp4) — file decode.
- `hound 3.5` — WAV reader and crash-safe append writer.
- `whisper-rs 0.16` (optional, default) — direct Whisper inference, pipes `initial_prompt`.
- `transcribe-rs 0.3` (onnx) — Parakeet wrapper.
- `reqwest 0.12` (rustls-tls, stream) + `sha2 0.10` + `futures-util 0.3` — model downloads with Range resume + SHA verify.
- `tokio 1` — `spawn_blocking` for inference and decode.
- `tracing 0.1` — backend boundary observability.
- `voice_activity_detector` / `silero-vad-rust` — **deferred** (ort 2.0.0-rc.10 vs 2.0.0-rc.12 conflict; see `audio-vad.md`).

Cargo feature matrix (`lumotia-transcription`):

| Feature | Default | Gates |
|---|---|---|
| `whisper` | yes | `whisper-rs` dep, `whisper_rs_backend` module, `load_whisper` fn |
| `whisper-vulkan` | yes | `whisper-rs/vulkan` (Vulkan GPU offload) |
| Parakeet (`transcribe-rs`) | always on | unconditional dep, no feature flag |

`lumotia-audio` has no Cargo features.

## Map of this slice

- [`audio-capture-pipeline.md`](audio-capture-pipeline.md) — `cpal` enumeration, monitor-source detection, RMS validation, hot-unplug error forwarding.
- [`audio-resampling.md`](audio-resampling.md) — `resample_to_16khz` (file) and `StreamingResampler` (live).
- [`audio-vad.md`](audio-vad.md) — `SpeechDetector` (current stub) and the ort version conflict that blocks Silero.
- [`audio-file-decoding.md`](audio-file-decoding.md) — `symphonia` decode, `decode_and_resample` async wrapper, `probe_audio_duration_secs`.
- [`audio-wav-io.md`](audio-wav-io.md) — `WavWriter` (crash-safe append) and `read_wav` / `write_wav` round-trip helpers.
- [`audio-pcm-bridge.md`](audio-pcm-bridge.md) — `static/pcm-processor.js` AudioWorklet (frontend-side fallback path).
- [`transcription-engines-overview.md`](transcription-engines-overview.md) — `Transcriber` trait, `TranscriberCapabilities`, `LocalEngine`.
- [`transcription-whisper.md`](transcription-whisper.md) — `WhisperRsBackend`, `WhisperContext`, params, `initial_prompt`, GPU offload.
- [`transcription-parakeet.md`](transcription-parakeet.md) — `SpeechModelAdapter` + `ParakeetWordGranularity`.
- [`transcription-streaming.md`](transcription-streaming.md) — `VadChunker` trait, `RmsVadChunker`, `LocalAgreement`, `trim_buffer_to_commit_point`.
- [`transcription-model-manager.md`](transcription-model-manager.md) — download flow, SHA verify, `.lumotia-verified` manifest, Range resume.
- [`transcription-concurrency.md`](transcription-concurrency.md) — `run_inference` async wrapper around `spawn_blocking`.
- [`cargo-features.md`](cargo-features.md) — feature matrix, build commands, rationale.
- [`build-tokenizers-guard.md`](build-tokenizers-guard.md) — `build.rs` Windows-MSVC-CRT guard against `tokenizers`.
- [`tests-and-fixtures.md`](tests-and-fixtures.md) — `thread_sweep.rs`, `whisper_rs_smoke.rs`, `jfk_bench.rs`, env-var gates, in-tree download server fixture.

## How this slice connects to others

- **Slice 2 (Tauri runtime)** owns `src-tauri/src/commands/{audio,transcription,live,models}.rs`. Those wrappers call into this slice via the `pub use` exports above. The live command in particular drives `MicrophoneCapture` + `StreamingResampler` + `RmsVadChunker` + `LocalAgreement` + `WavWriter` together. This crate publishes the primitives; slice 2 publishes the orchestrator.
- **Slice 4 (LLM + AI formatting)** runs after this slice produces a `Transcript`. It consumes the segment text via the storage layer, never directly. No type traffic flows back into this slice from formatting.
- **Slice 5 (core / storage / hotkey / build)** provides the shared types this slice depends on:
  - `lumotia_core::error::{MagnotiaError, Result}` — error envelope.
  - `lumotia_core::types::{AudioSamples, Segment, Transcript, TranscriptionOptions, EngineName, ModelId, DownloadProgress, Megabytes}`.
  - `lumotia_core::constants::{WHISPER_SAMPLE_RATE, VAD_SPEECH_THRESHOLD}`.
  - `lumotia_core::hardware::vulkan_loader_available` — runtime Vulkan probe used by `WhisperRsBackend`.
  - `lumotia_core::tuning::{inference_thread_count, Workload}` — power-aware thread count picker.
  - `lumotia_core::paths::app_paths` — `models_dir()` / `speech_model_dir()` resolution.
  - `lumotia_core::model_registry::{find_model, all_models, ModelEntry, ModelFile}` — declarative model catalogue.
  - Storage (slice 5) writes the produced `Transcript` to disk; this slice does not call into it directly.

## Open questions / debt

- **Silero VAD blocked on ort version conflict.** `crates/audio/src/vad.rs` is a stub that returns `true` for every input. Both `voice_activity_detector` and `silero-vad-rust` pin `ort 2.0.0-rc.10`; `transcribe-rs` (Parakeet) requires `2.0.0-rc.12`. The `RmsVadChunker` (transcription crate) is the live-capture fallback while this is unresolved.
- **VAD lives in two crates.** `audio::vad::SpeechDetector` is a single-frame predicate; `transcription::streaming::RmsVadChunker` is a stateful chunker. They share neither code nor thresholds. When Silero lands, expect consolidation.
- **Two resamplers maintained in parallel.** `resample::resample_to_16khz` (file) and `streaming_resample::StreamingResampler` (live) duplicate `SincInterpolationParameters` config. Drift between them will show up as audible mismatch between file imports and live captures.
- **Streaming primitives not yet wired into `live.rs`.** Comments in `streaming/mod.rs`, `buffer_trim.rs`, and `commit_policy.rs` flag that the integration into `src-tauri/src/commands/live.rs` ships as follow-up commits. Slice 2 is the place to verify whether that has landed.
- **`WhisperRsBackend` builds a fresh `WhisperState` per call.** Comment notes "state can be reused, but fresh-per-call is simpler". Worth measuring once we have the live-streaming path producing many small calls per session.
- **Parakeet quantisation is hardcoded to `Int8`** (`local_engine::load_parakeet`). No pathway for selecting a different quant.
- **Hardcoded `set_n_threads` in tests** (`jfk_bench.rs` uses 6) versus the production helper (`inference_thread_count(Workload::Whisper, gpu_offloaded)`). Test results are not comparable to runtime numbers without re-running with the helper-picked thread count.
- **`build.rs` panics at link time** on Windows if `tokenizers` ever enters the dep graph. This is intentional defence (Whispering v7.11.0 incident referenced in the file) but means a transitive pull of `tokenizers` from any sibling crate is a hard build break — not a soft warning. Worth surfacing in a top-level CONTRIBUTING note.
- **Symphonia feature list is fixed.** The `mp3, aac, flac, pcm, vorbis, wav, ogg, isomp4` set covers the import paths in v0.x. AAC requires patent-licensed code paths in some build configs — flagged by the brief.

## Existing in-repo docs (do not duplicate)

- `docs/whisper-ecosystem/brief.md` — top-level Whisper-ecosystem audit (the items #6, #8, #13, #19, #21, #24, #25, #26 referenced throughout this slice).
- `docs/whisper-ecosystem/workstream-A.md` — VAD / streaming roadmap (the source of `RmsVadChunker` thresholds).
- `docs/whisper-ecosystem/workstream-B.md` — UI commit/tentative contract that drives `LocalAgreement`.
- `docs/whisper-ecosystem/lumotia-context.md` — context summary for the workstreams.
- `docs/code-review-2026-04-22.md` — audit pass that flagged decode.rs RB-09 and `read_wav` filter_map bug (both fixed in tree).
- `docs/issues/decoder-partial-audio-on-error.md` — the RB-09 ticket.
- `docs/issues/native-capture-worker-join.md`, `c1-live-session-race.md`, `run-live-session-monolith.md` — slice 2's live-session debt; they reference primitives in this slice.
- `docs/brief/technology-map.md` — top-level tech map (covers all crates, not just this slice).