Jake
094b533ef2
dropped_chunks was incremented on cpal-callback channel-full and validation requeue overflow but never read by the live session, so the UI's dropped_audio_ms missed callback-level losses entirely. Architecture doc had flagged this as a TODO. Also: the 350ms validation buffer was requeued via try_send into the same 32-slot channel, silently dropping past the cap on small-buffer audio hosts (WASAPI exclusive, low-latency ALSA at 256 frames -> ~65 chunks). Fix: live runtime reads MicrophoneCapture::dropped_chunks() on each recv_audio tick (LiveSessionRuntime::poll_capture_drops) and converts the per-chunk-duration delta into the dropped_audio_ms surfaced to the UI overload status. Per-chunk duration is derived from the most recent AudioChunk's sample_rate + samples-per-channel so it adapts to whatever rate cpal is delivering at. Validation requeue moved from try_send into the bounded channel onto a VecDeque<AudioChunk> returned alongside the Receiver; ActiveCapture drains the replay buffer before reading rx in recv_audio, bypassing the 32-slot cap entirely. Architecture doc updated to remove the TODO and document the new pre-roll path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
121 lines
9.0 KiB
Markdown
121 lines
9.0 KiB
Markdown
---
|
|
name: Audio capture pipeline (cpal)
|
|
type: architecture-map-page
|
|
slice: 03-audio-transcription
|
|
last_verified: 2026/05/09
|
|
---
|
|
|
|
# Audio capture pipeline (cpal)
|
|
|
|
> **Where you are:** [Architecture map](../README.md) → [Audio + Transcription](README.md) → Audio capture pipeline
|
|
|
|
**Plain English summary.** The microphone-capture path enumerates host input devices, picks one (default first, then non-monitor sources, monitor sources only as a last resort), validates it produces real audio energy via a 350 ms RMS sniff, then opens a `cpal` stream that converts every supported sample format to `f32` and pushes `AudioChunk`s into a bounded `mpsc` channel. Runtime errors during the stream (mic unplug, audio server crash) are forwarded on a separate channel so the live session can show a toast.
|
|
|
|
## At a glance
|
|
|
|
- Crate: `lumotia-audio`
|
|
- Path: `crates/audio/src/capture.rs`
|
|
- LOC: 583
|
|
- External deps: `cpal 0.17`, `serde 1` (DeviceInfo wire type)
|
|
- Internal callers (best effort, slice 2 reconciles): `src-tauri/src/commands/audio.rs` (device list), `src-tauri/src/commands/live.rs` (start/stop, chunk + error channels).
|
|
|
|
Public surface:
|
|
|
|
- `pub struct AudioChunk` — `crates/audio/src/capture.rs:24` — `samples: Vec<f32>`, `sample_rate: u32`, `channels: u16`.
|
|
- `pub struct DeviceInfo` — `crates/audio/src/capture.rs:33` — Serde-derived for the Tauri IPC boundary.
|
|
- `pub struct CaptureRuntimeError` — `crates/audio/src/capture.rs:58` — non-fatal stream error reported after `start()` returned.
|
|
- `pub struct MicrophoneCapture` — `crates/audio/src/capture.rs:64`.
|
|
- `pub fn MicrophoneCapture::list_devices() -> Result<Vec<DeviceInfo>>` — `crates/audio/src/capture.rs:94`.
|
|
- `pub fn MicrophoneCapture::start_with_device(name: &str) -> Result<(Self, mpsc::Receiver<AudioChunk>)>` — `crates/audio/src/capture.rs:137`.
|
|
- `pub fn MicrophoneCapture::start() -> Result<(Self, mpsc::Receiver<AudioChunk>)>` — `crates/audio/src/capture.rs:166`.
|
|
- `pub fn MicrophoneCapture::stop(&mut self)` — `crates/audio/src/capture.rs:240`.
|
|
- `pub fn MicrophoneCapture::dropped_chunks(&self) -> u64` — `crates/audio/src/capture.rs:80`.
|
|
- `pub fn MicrophoneCapture::take_error_rx(&mut self) -> Option<mpsc::Receiver<CaptureRuntimeError>>` — `crates/audio/src/capture.rs:88`.
|
|
|
|
`MicrophoneCapture` implements `Drop` (`crates/audio/src/capture.rs:247`) so a panicked caller still pauses the cpal stream.
|
|
|
|
## What's in here
|
|
|
|
### Constants and tunables
|
|
|
|
- `AUDIO_CHANNEL_CAPACITY = 32` (`capture.rs:11`) — bounded capacity for the chunk channel.
|
|
- `DEVICE_VALIDATION_MS = 350` (`capture.rs:15`) — how long the sniffer listens before deciding a device is real.
|
|
- `SILENCE_RMS_FLOOR = 1e-5` (`capture.rs:21`) — lower bound for "produced real audio" during validation.
|
|
- `DEAD_SILENCE_FLOOR = 1e-7` (`capture.rs:475`) — even fallback (monitor) sources must clear this; pure dead-zero is rejected.
|
|
|
|
### Device selection (auto)
|
|
|
|
`start()` (`capture.rs:166`) sorts every input device into four tiers, tries each in order, and stops at the first that passes RMS validation:
|
|
|
|
1. Default + non-monitor (key 0).
|
|
2. Any other non-monitor (key 1).
|
|
3. Default but is a monitor (key 2, very rare).
|
|
4. Monitor source last resort (key 3).
|
|
|
|
The first pass enforces `require_audio = true`. If nothing clears the silence floor the loop does a second pass with `require_audio = false`, which still rejects dead silence but accepts a monitor source as a fallback.
|
|
|
|
Monitor detection (`is_monitor_name`, `capture.rs:258`) catches the standard PulseAudio / PipeWire patterns: `.monitor` suffix, `Monitor of ` prefix, anything containing `loopback`.
|
|
|
|
### Device selection (explicit)
|
|
|
|
`start_with_device()` (`capture.rs:137`) takes an exact device name (matched against `cpal::Device::description().name()`) and refuses to fall back. It still runs the same RMS validation. Returns a clear "open Settings → Audio" message if the requested device is no longer enumerated.
|
|
|
|
### ALSA description enrichment (Linux only)
|
|
|
|
`load_alsa_card_descriptions()` (`capture.rs:297`) parses `/proc/asound/cards` to map cpal's terse short name (`Microphones`) to the human-readable product string (`Blue Microphones`). Empty map on non-Linux. `extract_card_id()` (`capture.rs:278`) pulls `CARD=...` out of an ALSA device string.
|
|
|
|
### Stream construction
|
|
|
|
`open_and_validate()` (`capture.rs:351`) is the single entry point that:
|
|
|
|
1. Reads `default_input_config()` for sample rate, channel count, sample format.
|
|
2. Creates a `mpsc::sync_channel::<AudioChunk>(AUDIO_CHANNEL_CAPACITY)`.
|
|
3. Creates a `mpsc::sync_channel::<CaptureRuntimeError>(32)` for runtime errors.
|
|
4. Dispatches to `build_input_stream::<T>` for the device's sample format (F32 / I16 / U16). Anything else returns `MagnotiaError::AudioCaptureFailed`.
|
|
5. Calls `stream.play()`.
|
|
6. Sniffs samples for `DEVICE_VALIDATION_MS`, sums squared samples, derives RMS.
|
|
7. Rejects below floor (or dead silence). Otherwise hands the validation chunks back to the consumer as a `VecDeque` pre-roll alongside the live `mpsc::Receiver`, so downstream consumers do not lose the first 350 ms even on small-buffer hosts that would overflow the bounded channel.
|
|
|
|
`build_input_stream::<T>` (`capture.rs:505`) is generic over `T: Sample + SizedSample` with `f32: FromSample<T>` so the same body handles all three sample formats. The data callback maps every sample to `f32`, packages an `AudioChunk`, and `try_send`s on the channel; failure increments the `dropped_chunks` atomic. The error callback ships a `CaptureRuntimeError` on `err_tx` (channel-full path increments `dropped_errors` and logs to stderr).
|
|
|
|
### Drop counters
|
|
|
|
Two atomics give the live session visibility:
|
|
|
|
- `dropped_chunks: Arc<AtomicU64>` — chunk channel was full. Diagnostic for downstream backpressure.
|
|
- `dropped_errors: Arc<AtomicU64>` (private) — runtime-error channel was full (caller stopped draining). Logged to stderr each time.
|
|
|
|
## Data flow
|
|
|
|
```
|
|
cpal default_host()
|
|
└─ input_devices() (enumerate)
|
|
└─ open_and_validate(device)
|
|
├─ build_input_stream::<f32|i16|u16>(...)
|
|
│ └─ data callback: T → f32 → AudioChunk → mpsc::sync_channel
|
|
│ └─ error callback: cpal::StreamError → CaptureRuntimeError → err_mpsc
|
|
└─ stream.play()
|
|
└─ 350 ms sniff → RMS → accept | reject
|
|
└─ on accept: return MicrophoneCapture + VecDeque<AudioChunk> pre-roll + Receiver<AudioChunk>
|
|
```
|
|
|
|
Output: one `AudioChunk` per cpal callback period at the device's native rate. The live session is responsible for downmixing channels (if `channels > 1`) and feeding `StreamingResampler` to reach 16 kHz mono. Native rate is *not* normalised here.
|
|
|
|
## Watch-outs
|
|
|
|
- **Channel capacity is 32 chunks.** At a 1024-frame cpal buffer at 48 kHz that's roughly 700 ms. A blocked consumer for longer than that means dropped audio. `dropped_chunks()` is the visibility hook; the live-session command reads it on every `recv_audio` tick (`commands/live.rs::poll_capture_drops`) and converts the delta into the `dropped_audio_ms` surfaced to the UI overlay.
|
|
- **Default-device first works against the safest pick on Linux Pulse setups** where the default sink monitor sneaks in. The four-tier sort handles this, but only because monitor names match the patterns in `is_monitor_name`. New PipeWire schemes that don't include `.monitor` / `loopback` would slip through.
|
|
- **350 ms validation window adds a startup latency floor.** Slice 2 needs to know about this when wiring "click record".
|
|
- **`stop()` is `pause`, not `drop`.** The stream object is kept alive until `Drop`. A subsequent `start()` on the same `MicrophoneCapture` is not supported (signature returns a fresh instance).
|
|
- **Sample format dispatch is closed-set.** Anything not F32 / I16 / U16 is a hard error. cpal can in principle expose I8 / I32 / F64 on exotic devices.
|
|
- **`device_display_name` swallows errors.** `cpal::Device::description()` errors silently become `None`, then `<unnamed>` downstream. Acceptable for a UI list, surprising for debugging.
|
|
- **Validation pre-roll is returned out-of-band, not requeued.** `open_and_validate` returns a `VecDeque<AudioChunk>` alongside the live `mpsc::Receiver`; the live-session consumer drains the deque before reading the channel. This bypasses the 32-slot cap entirely so small-buffer hosts (WASAPI exclusive, low-latency ALSA at 256 frames) don't lose ~150ms from the head of every recording. Earlier versions used `try_send` to requeue and silently dropped half of the pre-roll on those hosts.
|
|
|
|
## See also
|
|
|
|
- [Audio resampling](audio-resampling.md) — the live session feeds `AudioChunk.samples` into `StreamingResampler` to get to 16 kHz.
|
|
- [Audio VAD](audio-vad.md) — what happens to chunks once they're at 16 kHz.
|
|
- [Audio WAV I/O](audio-wav-io.md) — `WavWriter` is the crash-safe sibling that persists capture audio to disk.
|
|
- [Audio PCM bridge](audio-pcm-bridge.md) — the static AudioWorklet that exists for browser-side fallback paths.
|
|
- Tests at `crates/audio/src/capture.rs:567` cover monitor pattern detection. The validation-window logic is exercised end-to-end through the live integration tests in slice 2.
|