jake/Lumotia
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

audio: fix mic capture — skip monitor sources, validate by RMS, add device picker

Browse Source 
Day 1 of the upgrade plan (output/reports/kon-upgrade-plan-2026-04-17.md
in the CORBEL workspace). Fixes the HANDOVER.md blocker: native live
transcription was capturing silence because PulseAudio/PipeWire monitor
sources (speaker loopback) were winning the device-selection race —
they deliver zero-valued bytes that satisfied the original
"any device that produces data within 350ms" check.

WHAT CHANGED:

crates/audio/src/capture.rs (rewrite):
- New `DeviceInfo` struct (serde-derived) for the Settings device picker
- New `MicrophoneCapture::list_devices()` enumerates inputs with metadata
- New `MicrophoneCapture::start_with_device(name)` for explicit selection
- Refactored `start()` with monitor-source filtering by name pattern
  (.monitor suffix, "Monitor of " prefix, "loopback" substring) and
  RMS-energy validation in a 350ms window
- Two-pass selection: real inputs first, monitor sources only as
  last-resort fallback with explicit warning log
- Drop counter (Arc<AtomicU64>) tracks chunks dropped by `try_send`
  failure under load — Codex review caught this as a silent-failure risk
- `dropped_chunks()` accessor for the live session
- Verbose tracing at every step for diagnosis
- Unit test for monitor-name detection
- `cargo check -p kon-audio` passes clean

crates/audio/src/lib.rs:
- Re-export `DeviceInfo`

crates/audio/Cargo.toml:
- Add serde dependency (for DeviceInfo derives)

src-tauri/src/commands/audio.rs:
- New `list_audio_devices` Tauri command (returns Vec<DeviceInfo>)

src-tauri/src/lib.rs:
- Register `list_audio_devices` in invoke_handler

src/lib/pages/SettingsPage.svelte:
- New "Audio" section with microphone picker dropdown
- Auto-populates on mount via `list_audio_devices`
- Refresh button + clear messaging about monitor sources
- Likely-monitor entries marked disabled in the dropdown
- Auto mode is the default (empty string in settings.microphoneDevice)

src/lib/stores/page.svelte.js:
- New `microphoneDevice` field in defaults (empty = auto-select)

NEXT STEPS (per the upgrade plan):
- Wire `microphoneDevice` from settings into `MicrophoneCapture::start_with_device`
  in the live and standalone capture paths (currently both still call
  the auto-selecting `start()`)
- Test on real hardware (Wayland + multiple input devices)
- Codex sanity-check of this diff is running in parallel; addendum to
  follow if anything substantive comes back

Refs: /home/jake/Documents/CORBEL-Projects/kon/HANDOVER.md
      output/reports/kon-upgrade-plan-2026-04-17.md (CORBEL workspace)
This commit is contained in:
Jake
2026-04-17 12:43:13 +01:00
parent fdc4a3cba5
commit 96980c7d5c
7 changed files with 805 additions and 103 deletions
Download Patch File Download Diff File Expand all files Collapse all files

370
crates/audio/src/capture.rs
View File

@@ -1,9 +1,25 @@
use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::mpsc;
use cpal::{FromSample, Sample, SampleFormat, SizedSample};
use cpal::traits::{DeviceTrait, HostTrait, StreamTrait};
use serde::{Deserialize, Serialize};
use kon_core::error::{KonError, Result};
const AUDIO_CHANNEL_CAPACITY: usize = 32;
/// Validation window. We listen for this long and compute RMS to decide
/// whether the chosen device is delivering real audio (vs a silent monitor).
const DEVICE_VALIDATION_MS: u64 = 350;
/// Below this RMS amplitude (peak ±1.0 scale) the input is treated as
/// silence. PulseAudio/PipeWire monitor sources for an idle speaker
/// typically deliver dead-zero samples; real microphones yield ~0.0005+
/// even in a quiet room. Conservative floor: 1e-5.
const SILENCE_RMS_FLOOR: f32 = 1e-5;
/// A chunk of captured audio from the microphone.
pub struct AudioChunk {
pub samples: Vec<f32>,
@@ -11,53 +27,180 @@ pub struct AudioChunk {
pub channels: u16,
}
/// Public-facing description of an audio input device.
/// Returned by `list_devices()` and used by the UI device picker.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct DeviceInfo {
/// Device name as reported by cpal/the host.
pub name: String,
/// Default sample rate in Hz.
pub sample_rate: u32,
/// Default channel count.
pub channels: u16,
/// True if the device name matches a known monitor-source pattern
/// (PulseAudio/PipeWire loopback of speaker output).
pub is_likely_monitor: bool,
/// True if cpal reports this as the host's default input device.
pub is_default: bool,
}
/// Manages microphone capture via cpal.
/// Call `start()` to begin capturing, which returns a receiver for audio chunks.
/// Call `stop()` to end the stream.
pub struct MicrophoneCapture {
stream: Option<cpal::Stream>,
/// Name of the device that is actually capturing.
pub device_name: String,
/// Counter incremented every time the capture callback drops a chunk
/// because the channel was full. Read via `dropped_chunks()`.
dropped_chunks: Arc<AtomicU64>,
}
impl MicrophoneCapture {
/// Start capturing audio from the default input device.
/// Returns a receiver that yields AudioChunks as they arrive.
/// Number of audio chunks dropped because the downstream channel was full
/// since this capture started. Should stay at 0 in normal use; non-zero
/// indicates downstream backpressure or a stuck consumer.
pub fn dropped_chunks(&self) -> u64 {
self.dropped_chunks.load(Ordering::Relaxed)
}
/// Enumerate every input device the host knows about, with the metadata
/// needed by the device-picker UI.
pub fn list_devices() -> Result<Vec<DeviceInfo>> {
let host = cpal::default_host();
let default_name = host
.default_input_device()
.and_then(|d| d.name().ok())
.unwrap_or_default();
let devices = host
.input_devices()
.map_err(|e| KonError::AudioCaptureFailed(format!("input_devices: {e}")))?;
let mut out = Vec::new();
for device in devices {
let name = device.name().unwrap_or_else(|_| "<unnamed>".to_string());
let (sample_rate, channels) = match device.default_input_config() {
Ok(cfg) => (cfg.sample_rate(), cfg.channels() as u16),
Err(_) => (0, 0),
};
let is_likely_monitor = is_monitor_name(&name);
let is_default = !default_name.is_empty() && name == default_name;
out.push(DeviceInfo {
name,
sample_rate,
channels,
is_likely_monitor,
is_default,
});
}
Ok(out)
}
/// Start capturing from the device whose name matches `device_name` exactly.
/// If no match is found, returns an error rather than silently falling back.
pub fn start_with_device(
device_name: &str,
) -> Result<(Self, mpsc::Receiver<AudioChunk>)> {
let host = cpal::default_host();
let devices = host
.input_devices()
.map_err(|e| KonError::AudioCaptureFailed(format!("input_devices: {e}")))?;
for device in devices {
let name = device.name().unwrap_or_default();
if name == device_name {
eprintln!("[kon-audio] start_with_device: opening explicit device '{name}'");
return open_and_validate(device, &name, /* require_audio = */ true);
}
}
Err(KonError::AudioCaptureFailed(format!(
"Selected device '{device_name}' not found in current host enumeration. \
It may have been disconnected. Open Settings → Audio to pick another."
)))
}
/// Start capturing audio with auto-selection.
///
/// Selection rules:
/// 1. Try the host default input device first if it exists AND is not a monitor source.
/// 2. Otherwise, try non-monitor devices in enumeration order.
/// 3. Validate the chosen device by RMS energy (not just receipt of bytes) over
/// a short window — this is what defeats the "silent monitor source wins" bug.
/// 4. If no non-monitor device produces real audio, fall back to monitor sources
/// as a last resort (with a clear log line). Never accept dead silence.
pub fn start() -> Result<(Self, mpsc::Receiver<AudioChunk>)> {
let host = cpal::default_host();
let device = host.default_input_device().ok_or_else(|| {
KonError::AudioCaptureFailed("No input device found".into())
})?;
let default_name = host
.default_input_device()
.and_then(|d| d.name().ok())
.unwrap_or_default();
let config = device.default_input_config().map_err(|e| {
KonError::AudioCaptureFailed(format!("No input config: {e}"))
})?;
let mut all_devices: Vec<cpal::Device> =
host.input_devices()
.map_err(|e| {
KonError::AudioCaptureFailed(format!("input_devices: {e}"))
})?
.collect();
let sample_rate = config.sample_rate();
let channels = config.channels() as u16;
// Sort: default first, then non-monitor, then monitor-as-last-resort.
all_devices.sort_by_key(|d| {
let n = d.name().unwrap_or_default();
let is_default = !default_name.is_empty() && n == default_name;
let is_monitor = is_monitor_name(&n);
// Smaller key = tried first.
match (is_default, is_monitor) {
(true, false) => 0, // default, real input
(false, false) => 1, // any other real input
(true, true) => 2, // default but is a monitor (very rare)
(false, true) => 3, // monitor source — last resort
}
});
let (tx, rx) = mpsc::channel::<AudioChunk>();
eprintln!(
"[kon-audio] start: enumerated {} input device(s) (default='{}')",
all_devices.len(),
default_name
);
let stream = device
.build_input_stream(
&config.into(),
move |data: &[f32], _info: &cpal::InputCallbackInfo| {
let _ = tx.send(AudioChunk {
samples: data.to_vec(),
sample_rate,
channels,
});
},
|err| eprintln!("audio capture error: {err}"),
None,
)
.map_err(|e| {
KonError::AudioCaptureFailed(format!("Build stream failed: {e}"))
})?;
// First pass: require real audio energy.
for device in &all_devices {
let name = device.name().unwrap_or_default();
if is_monitor_name(&name) {
continue; // Save monitor sources for second pass.
}
match open_and_validate(device.clone(), &name, true) {
Ok(result) => return Ok(result),
Err(e) => {
eprintln!("[kon-audio] '{name}' rejected: {e}");
}
}
}
stream.play().map_err(|e| {
KonError::AudioCaptureFailed(format!("Stream play failed: {e}"))
})?;
// Second pass: accept anything that delivers bytes (monitor sources
// included). Better to capture from a monitor than fail entirely.
eprintln!(
"[kon-audio] no non-monitor mic produced audio; falling back to monitor/loopback sources"
);
for device in &all_devices {
let name = device.name().unwrap_or_default();
match open_and_validate(device.clone(), &name, false) {
Ok(result) => {
eprintln!(
"[kon-audio] FALLBACK: capturing from '{name}' (likely monitor source). \
Recordings may be silent or contain system audio."
);
return Ok(result);
}
Err(_) => continue,
}
}
Ok((Self { stream: Some(stream) }, rx))
Err(KonError::AudioCaptureFailed(
"No working microphone found. Check that an input device is connected, \
that PulseAudio/PipeWire is running, and that the app has microphone permission. \
Then open Settings → Audio to pick a device explicitly."
.into(),
))
}
/// Stop capturing audio.
@@ -73,3 +216,164 @@ impl Drop for MicrophoneCapture {
self.stop();
}
}
/// Heuristic: identify a PulseAudio/PipeWire monitor source by name.
/// Common patterns:
/// - ".monitor" suffix (PulseAudio convention)
/// - "Monitor of " prefix (longer human-readable name)
/// - "Loopback" anywhere (some PipeWire configurations)
fn is_monitor_name(name: &str) -> bool {
let lower = name.to_lowercase();
lower.ends_with(".monitor")
|| lower.starts_with("monitor of ")
|| lower.contains("monitor of ")
|| lower.contains("loopback")
}
/// Open the given device and validate it produces non-silent audio.
/// If `require_audio` is false, accept any data (used for monitor fallback).
fn open_and_validate(
device: cpal::Device,
name: &str,
require_audio: bool,
) -> Result<(MicrophoneCapture, mpsc::Receiver<AudioChunk>)> {
let config = device.default_input_config().map_err(|e| {
KonError::AudioCaptureFailed(format!("default_input_config: {e}"))
})?;
let sample_rate = config.sample_rate();
let channels = config.channels() as u16;
let format = config.sample_format();
eprintln!(
"[kon-audio] trying '{name}' ({sr}Hz, {ch}ch, {fmt:?})",
sr = sample_rate,
ch = channels,
fmt = format
);
let (tx, rx) = mpsc::sync_channel::<AudioChunk>(AUDIO_CHANNEL_CAPACITY);
let requeue_tx = tx.clone();
let dropped_chunks = Arc::new(AtomicU64::new(0));
let stream = match format {
SampleFormat::F32 => build_input_stream::<f32>(&device, &config, sample_rate, channels, tx, dropped_chunks.clone()),
SampleFormat::I16 => build_input_stream::<i16>(&device, &config, sample_rate, channels, tx, dropped_chunks.clone()),
SampleFormat::U16 => build_input_stream::<u16>(&device, &config, sample_rate, channels, tx, dropped_chunks.clone()),
other => {
return Err(KonError::AudioCaptureFailed(format!(
"unsupported sample format {other:?}"
)))
}
}
.map_err(|e| KonError::AudioCaptureFailed(format!("build_input_stream: {e}")))?;
stream
.play()
.map_err(|e| KonError::AudioCaptureFailed(format!("stream.play: {e}")))?;
// Validation window: collect chunks for DEVICE_VALIDATION_MS, compute RMS.
let deadline = std::time::Instant::now()
+ std::time::Duration::from_millis(DEVICE_VALIDATION_MS);
let mut collected: Vec<AudioChunk> = Vec::new();
let mut total_samples = 0_usize;
let mut sum_sq: f64 = 0.0;
while std::time::Instant::now() < deadline {
let remaining = deadline.saturating_duration_since(std::time::Instant::now());
if remaining.is_zero() {
break;
}
match rx.recv_timeout(remaining) {
Ok(chunk) => {
for &s in &chunk.samples {
sum_sq += (s as f64) * (s as f64);
}
total_samples += chunk.samples.len();
collected.push(chunk);
}
Err(_) => break,
}
}
if total_samples == 0 {
return Err(KonError::AudioCaptureFailed(
"device delivered zero samples in validation window".into(),
));
}
let rms = (sum_sq / total_samples as f64).sqrt() as f32;
eprintln!(
"[kon-audio] '{name}' validation: {samples} samples, rms={rms:.6}",
samples = total_samples
);
if require_audio && rms < SILENCE_RMS_FLOOR {
return Err(KonError::AudioCaptureFailed(format!(
"device produced silence (rms={rms:.6} below floor {SILENCE_RMS_FLOOR:.6})"
)));
}
// Re-queue the collected chunks so downstream gets them.
for chunk in collected {
let _ = requeue_tx.try_send(chunk);
}
eprintln!("[kon-audio] selected microphone: '{name}'");
Ok((
MicrophoneCapture {
stream: Some(stream),
device_name: name.to_string(),
dropped_chunks,
},
rx,
))
}
fn build_input_stream<T>(
device: &cpal::Device,
supported_config: &cpal::SupportedStreamConfig,
sample_rate: u32,
channels: u16,
tx: mpsc::SyncSender<AudioChunk>,
dropped_chunks: Arc<AtomicU64>,
) -> std::result::Result<cpal::Stream, cpal::BuildStreamError>
where
T: Sample + SizedSample,
f32: FromSample<T>,
{
let config: cpal::StreamConfig = supported_config.clone().into();
device.build_input_stream(
&config,
move |data: &[T], _| {
let samples: Vec<f32> = data.iter().copied().map(f32::from_sample).collect();
let chunk = AudioChunk {
samples,
sample_rate,
channels,
};
// try_send fails if the channel is full. Track that explicitly
// rather than swallowing it — Codex review 2026/04/17 caught
// this as a silent-failure risk under sustained load.
if tx.try_send(chunk).is_err() {
dropped_chunks.fetch_add(1, Ordering::Relaxed);
}
},
|err| eprintln!("[kon-audio] capture error: {err}"),
None,
)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn monitor_pattern_detection() {
assert!(is_monitor_name("alsa_output.pci-0000_00_1f.3.analog-stereo.monitor"));
assert!(is_monitor_name("Monitor of Built-in Audio Analog Stereo"));
assert!(is_monitor_name("Some Loopback Device"));
assert!(!is_monitor_name("Blue Yeti USB"));
assert!(!is_monitor_name("alsa_input.pci-0000_00_1f.3.analog-stereo"));
assert!(!is_monitor_name(""));
}
}