Files
Lumotia/crates/audio/src/capture.rs
Jake 34fce3cf9e
Some checks failed
check / cargo check (macos-latest) (push) Has been cancelled
check / cargo check (ubuntu-22.04) (push) Has been cancelled
check / cargo check (windows-latest) (push) Has been cancelled
check / svelte build + lint (push) Has been cancelled
feat: OpenWhispr-inspired transcription polish pass
Major quality pass on top of Phase 2. Five substantive changes plus
cross-cutting touches across audio, hotkey, transcription, and Tauri
command layers.

  Transcription quality

  - Long-audio chunking in commands/transcription.rs: Parakeet and large
    file transcription now chunk-and-recompose with overlap trimming, so
    the live-path chunking advantage extends to file-based workflows.
  - Stateful live speech gate in commands/live.rs on top of the earlier
    duplicate-boundary filtering — distinguishes start-of-speech from
    mid-speech and holds state across chunks.

  Auto-learning corrections

  - New crates/ai-formatting/src/correction_learning.rs: extracts user
    text corrections from viewer edits and proposes additions to the
    active profile's vocabulary.
  - src-tauri/src/commands/profiles.rs bridge for frontend-driven
    confirmation of learned terms.
  - src/routes/viewer/+page.svelte hooks the learning path into the
    segment-edit flow so corrections feed profile_terms without a
    separate 'train this profile' UX.

  Transcript profile provenance

  - Migration v8 (crates/storage/src/migrations.rs) adds profile_id to
    transcripts, defaulting to DEFAULT_PROFILE_ID so existing rows stay
    valid.
  - crates/storage/src/database.rs: TranscriptRow + CRUD carry profile_id.
  - src-tauri/src/commands/transcripts.rs: add_transcript accepts and
    persists profile_id.
  - DictationPage.svelte + FilesPage.svelte send activeProfileId on
    capture so learned corrections are attributed to the right profile.

  Cleanup prompt contract

  - crates/ai-formatting/src/llm_client.rs hardened: the CLEANUP_PROMPT
    now specifies concrete do/do-not rules, ready for a real model-backed
    cleanup pass. The llm_client is still a stub — kon-llm remains unwired
    — but the prompt shape is final.

  Cross-cutting polish

  - Minor touches in audio (capture/decode/resample), hotkey (lib/linux/stub),
    core, transcription (concurrency/model_manager/local_engine/whisper_rs),
    and the rest of src-tauri/src/commands/*: error-path tightening, log
    clarity, TS-migration follow-ups (@ts-nocheck additions for incremental
    typing).

Verified locally: npm run check, cargo test -p kon-ai-formatting,
cargo test -p kon-storage, cargo test -p kon --lib commands::live::tests,
cargo check — all green.

Scope boundary: kon-llm crate is still a stub; task extraction remains
rule-based. Bundled local-LLM runtime is the next clean step and is not
in this commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 22:39:08 +01:00

566 lines
21 KiB
Rust

use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::mpsc;
use std::sync::Arc;
use cpal::traits::{DeviceTrait, HostTrait, StreamTrait};
use cpal::{FromSample, Sample, SampleFormat, SizedSample};
use serde::{Deserialize, Serialize};
use kon_core::error::{KonError, Result};
const AUDIO_CHANNEL_CAPACITY: usize = 32;
/// Validation window. We listen for this long and compute RMS to decide
/// whether the chosen device is delivering real audio (vs a silent monitor).
const DEVICE_VALIDATION_MS: u64 = 350;
/// Below this RMS amplitude (peak ±1.0 scale) the input is treated as
/// silence. PulseAudio/PipeWire monitor sources for an idle speaker
/// typically deliver dead-zero samples; real microphones yield ~0.0005+
/// even in a quiet room. Conservative floor: 1e-5.
const SILENCE_RMS_FLOOR: f32 = 1e-5;
/// A chunk of captured audio from the microphone.
pub struct AudioChunk {
pub samples: Vec<f32>,
pub sample_rate: u32,
pub channels: u16,
}
/// Public-facing description of an audio input device.
/// Returned by `list_devices()` and used by the UI device picker.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct DeviceInfo {
/// Device name as reported by cpal/the host.
pub name: String,
/// Default sample rate in Hz.
pub sample_rate: u32,
/// Default channel count.
pub channels: u16,
/// True if the device name matches a known monitor-source pattern
/// (PulseAudio/PipeWire loopback of speaker output).
pub is_likely_monitor: bool,
/// True if cpal reports this as the host's default input device.
pub is_default: bool,
/// Human-readable product description, if known (Linux: from
/// `/proc/asound/cards`). Empty string when unavailable or on
/// platforms that don't expose one.
#[serde(default)]
pub description: String,
}
/// A non-fatal capture-time error emitted by the cpal stream callback after
/// `start()` has already returned. The live session subscribes to these via
/// `error_rx()` so the frontend can show a toast when the mic vanishes
/// mid-recording.
/// (Codex review 2026/04/17 M2)
#[derive(Debug, Clone)]
pub struct CaptureRuntimeError {
pub device_name: String,
pub message: String,
}
/// Manages microphone capture via cpal.
pub struct MicrophoneCapture {
stream: Option<cpal::Stream>,
/// Name of the device that is actually capturing.
pub device_name: String,
/// Counter incremented every time the capture callback drops a chunk
/// because the channel was full. Read via `dropped_chunks()`.
dropped_chunks: Arc<AtomicU64>,
/// Receiver for runtime stream errors (device unplugged, audio server
/// crash, etc.). The live session calls `error_rx()` once and listens.
error_rx: Option<mpsc::Receiver<CaptureRuntimeError>>,
}
impl MicrophoneCapture {
/// Number of audio chunks dropped because the downstream channel was full
/// since this capture started. Should stay at 0 in normal use; non-zero
/// indicates downstream backpressure or a stuck consumer.
pub fn dropped_chunks(&self) -> u64 {
self.dropped_chunks.load(Ordering::Relaxed)
}
/// Take the runtime-error receiver. Can be called once per capture; the
/// caller (live session manager) drains it on its own cadence and surfaces
/// errors to the frontend. Returns None on the second call.
/// (Codex review 2026/04/17 M2)
pub fn take_error_rx(&mut self) -> Option<mpsc::Receiver<CaptureRuntimeError>> {
self.error_rx.take()
}
/// Enumerate every input device the host knows about, with the metadata
/// needed by the device-picker UI.
pub fn list_devices() -> Result<Vec<DeviceInfo>> {
let host = cpal::default_host();
let default_name = host
.default_input_device()
.and_then(|d| device_display_name(&d))
.unwrap_or_default();
let devices = host
.input_devices()
.map_err(|e| KonError::AudioCaptureFailed(format!("input_devices: {e}")))?;
// Load ALSA card descriptions once per enumeration. These are the
// "real" product names (e.g. "Blue Microphones") that cpal's
// short card name (e.g. "Microphones") alone can't convey. Empty
// map on non-Linux or if the file is missing.
let card_descriptions = load_alsa_card_descriptions();
let mut out = Vec::new();
for device in devices {
let name = device_display_name(&device).unwrap_or_else(|| "<unnamed>".to_string());
let (sample_rate, channels) = match device.default_input_config() {
Ok(cfg) => (cfg.sample_rate(), cfg.channels() as u16),
Err(_) => (0, 0),
};
let is_likely_monitor = is_monitor_name(&name);
let is_default = !default_name.is_empty() && name == default_name;
let description = extract_card_id(&name)
.and_then(|card| card_descriptions.get(card).cloned())
.unwrap_or_default();
out.push(DeviceInfo {
name,
sample_rate,
channels,
is_likely_monitor,
is_default,
description,
});
}
Ok(out)
}
/// Start capturing from the device whose name matches `device_name` exactly.
/// If no match is found, returns an error rather than silently falling back.
pub fn start_with_device(device_name: &str) -> Result<(Self, mpsc::Receiver<AudioChunk>)> {
let host = cpal::default_host();
let devices = host
.input_devices()
.map_err(|e| KonError::AudioCaptureFailed(format!("input_devices: {e}")))?;
for device in devices {
let name = device_display_name(&device).unwrap_or_default();
if name == device_name {
eprintln!("[kon-audio] start_with_device: opening explicit device '{name}'");
return open_and_validate(device, &name, /* require_audio = */ true);
}
}
Err(KonError::AudioCaptureFailed(format!(
"Selected device '{device_name}' not found in current host enumeration. \
It may have been disconnected. Open Settings → Audio to pick another."
)))
}
/// Start capturing audio with auto-selection.
///
/// Selection rules:
/// 1. Try the host default input device first if it exists AND is not a monitor source.
/// 2. Otherwise, try non-monitor devices in enumeration order.
/// 3. Validate the chosen device by RMS energy (not just receipt of bytes) over
/// a short window — this is what defeats the "silent monitor source wins" bug.
/// 4. If no non-monitor device produces real audio, fall back to monitor sources
/// as a last resort (with a clear log line). Never accept dead silence.
pub fn start() -> Result<(Self, mpsc::Receiver<AudioChunk>)> {
let host = cpal::default_host();
let default_name = host
.default_input_device()
.and_then(|d| device_display_name(&d))
.unwrap_or_default();
let mut all_devices: Vec<cpal::Device> = host
.input_devices()
.map_err(|e| KonError::AudioCaptureFailed(format!("input_devices: {e}")))?
.collect();
// Sort: default first, then non-monitor, then monitor-as-last-resort.
all_devices.sort_by_key(|d| {
let n = device_display_name(d).unwrap_or_default();
let is_default = !default_name.is_empty() && n == default_name;
let is_monitor = is_monitor_name(&n);
// Smaller key = tried first.
match (is_default, is_monitor) {
(true, false) => 0, // default, real input
(false, false) => 1, // any other real input
(true, true) => 2, // default but is a monitor (very rare)
(false, true) => 3, // monitor source — last resort
}
});
eprintln!(
"[kon-audio] start: enumerated {} input device(s) (default='{}')",
all_devices.len(),
default_name
);
// First pass: require real audio energy.
for device in &all_devices {
let name = device_display_name(device).unwrap_or_default();
if is_monitor_name(&name) {
continue; // Save monitor sources for second pass.
}
match open_and_validate(device.clone(), &name, true) {
Ok(result) => return Ok(result),
Err(e) => {
eprintln!("[kon-audio] '{name}' rejected: {e}");
}
}
}
// Second pass: accept anything that delivers bytes (monitor sources
// included). Better to capture from a monitor than fail entirely.
eprintln!(
"[kon-audio] no non-monitor mic produced audio; falling back to monitor/loopback sources"
);
for device in &all_devices {
let name = device_display_name(device).unwrap_or_default();
match open_and_validate(device.clone(), &name, false) {
Ok(result) => {
eprintln!(
"[kon-audio] FALLBACK: capturing from '{name}' (likely monitor source). \
Recordings may be silent or contain system audio."
);
return Ok(result);
}
Err(_) => continue,
}
}
Err(KonError::AudioCaptureFailed(
"No working microphone found. Check that an input device is connected, \
that PulseAudio/PipeWire is running, and that the app has microphone permission. \
Then open Settings → Audio to pick a device explicitly."
.into(),
))
}
/// Stop capturing audio.
pub fn stop(&mut self) {
if let Some(stream) = self.stream.take() {
let _ = stream.pause();
}
}
}
impl Drop for MicrophoneCapture {
fn drop(&mut self) {
self.stop();
}
}
/// Heuristic: identify a PulseAudio/PipeWire monitor source by name.
/// Common patterns:
/// - ".monitor" suffix (PulseAudio convention)
/// - "Monitor of " prefix (longer human-readable name)
/// - "Loopback" anywhere (some PipeWire configurations)
fn is_monitor_name(name: &str) -> bool {
let lower = name.to_lowercase();
lower.ends_with(".monitor")
|| lower.starts_with("monitor of ")
|| lower.contains("monitor of ")
|| lower.contains("loopback")
}
fn device_display_name(device: &cpal::Device) -> Option<String> {
device
.description()
.ok()
.map(|description| description.name().to_string())
}
/// Pull the CARD= value from an ALSA device string.
///
/// `sysdefault:CARD=Microphones` → `Some("Microphones")`
/// `hw:CARD=C920,DEV=0` → `Some("C920")`
/// `pipewire` / `default` → `None`
fn extract_card_id(name: &str) -> Option<&str> {
let rest = name.split("CARD=").nth(1)?;
Some(
rest.split(|c: char| c == ',' || c == ';')
.next()
.unwrap_or(rest),
)
}
/// Read `/proc/asound/cards` and return a map from ALSA card short name
/// (e.g. "Microphones") to the richer product string (e.g. "Blue
/// Microphones"). Empty map on non-Linux or if the file is missing.
///
/// Format of `/proc/asound/cards`:
/// ```text
/// 2 [Microphones ]: USB-Audio - Blue Microphones
/// Blue Microphones at usb-...
/// 3 [C920 ]: USB-Audio - HD Pro Webcam C920
/// HD Pro Webcam C920 at usb-...
/// ```
/// The bracket contains the short name that cpal reports; the text
/// after the colon on that same line is the description we want. The
/// next indented line is a longer location string we ignore.
fn load_alsa_card_descriptions() -> std::collections::HashMap<String, String> {
use std::collections::HashMap;
let mut map = HashMap::new();
#[cfg(target_os = "linux")]
{
let Ok(contents) = std::fs::read_to_string("/proc/asound/cards") else {
return map;
};
for line in contents.lines() {
// Header lines start with an optional leading space plus a
// digit (the card ID, right-aligned to 2 chars for readable
// formatting). Continuation lines are indented beyond that.
let trimmed = line.trim_start();
if !trimmed
.chars()
.next()
.map(|c| c.is_ascii_digit())
.unwrap_or(false)
{
continue;
}
let Some(open) = trimmed.find('[') else {
continue;
};
let Some(close) = trimmed[open..].find(']') else {
continue;
};
let short_name = trimmed[open + 1..open + close].trim().to_string();
if short_name.is_empty() {
continue;
}
let after_bracket = &trimmed[open + close + 1..];
let Some(colon) = after_bracket.find(':') else {
continue;
};
// Format: "USB-Audio - Blue Microphones"
// We keep everything after the " - " if present, otherwise
// the whole post-colon fragment.
let raw = after_bracket[colon + 1..].trim();
let description = raw
.split(" - ")
.nth(1)
.map(|s| s.trim().to_string())
.unwrap_or_else(|| raw.to_string());
if !description.is_empty() {
map.insert(short_name, description);
}
}
}
map
}
/// Open the given device and validate it produces non-silent audio.
/// If `require_audio` is false, accept any data (used for monitor fallback).
fn open_and_validate(
device: cpal::Device,
name: &str,
require_audio: bool,
) -> Result<(MicrophoneCapture, mpsc::Receiver<AudioChunk>)> {
let config = device
.default_input_config()
.map_err(|e| KonError::AudioCaptureFailed(format!("default_input_config: {e}")))?;
let sample_rate = config.sample_rate();
let channels = config.channels() as u16;
let format = config.sample_format();
eprintln!(
"[kon-audio] trying '{name}' ({sr}Hz, {ch}ch, {fmt:?})",
sr = sample_rate,
ch = channels,
fmt = format
);
let (tx, rx) = mpsc::sync_channel::<AudioChunk>(AUDIO_CHANNEL_CAPACITY);
let requeue_tx = tx.clone();
let dropped_chunks = Arc::new(AtomicU64::new(0));
// Bounded channel for runtime stream errors. Capacity 16 = plenty for
// the rare error case; if it ever fills, we drop newer errors silently
// because they would be redundant noise in a stream that is already
// failing. (Codex review 2026/04/17 M2)
let (err_tx, err_rx) = mpsc::sync_channel::<CaptureRuntimeError>(16);
let stream = match format {
SampleFormat::F32 => build_input_stream::<f32>(
&device,
&config,
sample_rate,
channels,
tx,
dropped_chunks.clone(),
err_tx.clone(),
name.to_string(),
),
SampleFormat::I16 => build_input_stream::<i16>(
&device,
&config,
sample_rate,
channels,
tx,
dropped_chunks.clone(),
err_tx.clone(),
name.to_string(),
),
SampleFormat::U16 => build_input_stream::<u16>(
&device,
&config,
sample_rate,
channels,
tx,
dropped_chunks.clone(),
err_tx.clone(),
name.to_string(),
),
other => {
return Err(KonError::AudioCaptureFailed(format!(
"unsupported sample format {other:?}"
)))
}
}
.map_err(|e| KonError::AudioCaptureFailed(format!("build_input_stream: {e}")))?;
stream
.play()
.map_err(|e| KonError::AudioCaptureFailed(format!("stream.play: {e}")))?;
// Validation window: collect chunks for DEVICE_VALIDATION_MS, compute RMS.
let deadline =
std::time::Instant::now() + std::time::Duration::from_millis(DEVICE_VALIDATION_MS);
let mut collected: Vec<AudioChunk> = Vec::new();
let mut total_samples = 0_usize;
let mut sum_sq: f64 = 0.0;
while std::time::Instant::now() < deadline {
let remaining = deadline.saturating_duration_since(std::time::Instant::now());
if remaining.is_zero() {
break;
}
match rx.recv_timeout(remaining) {
Ok(chunk) => {
for &s in &chunk.samples {
sum_sq += (s as f64) * (s as f64);
}
total_samples += chunk.samples.len();
collected.push(chunk);
}
Err(_) => break,
}
}
if total_samples == 0 {
return Err(KonError::AudioCaptureFailed(
"device delivered zero samples in validation window".into(),
));
}
let rms = (sum_sq / total_samples as f64).sqrt() as f32;
eprintln!(
"[kon-audio] '{name}' validation: {samples} samples, rms={rms:.6}",
samples = total_samples
);
if require_audio && rms < SILENCE_RMS_FLOOR {
return Err(KonError::AudioCaptureFailed(format!(
"device produced silence (rms={rms:.6} below floor {SILENCE_RMS_FLOOR:.6})"
)));
}
// Even in the fallback pass (require_audio=false), reject completely
// dead-zero audio. PulseAudio/PipeWire will sometimes happily emit a
// long stream of f32 zeros from a borked device — that is worse than
// failing fast. (Codex review 2026/04/17 D3)
const DEAD_SILENCE_FLOOR: f32 = 1e-7;
if rms < DEAD_SILENCE_FLOOR {
return Err(KonError::AudioCaptureFailed(format!(
"device produced dead silence (rms={rms:.6e} below absolute floor {DEAD_SILENCE_FLOOR:.6e})"
)));
}
// Re-queue the collected chunks so downstream gets them. Count any
// drops here against the same `dropped_chunks` counter so the live
// session sees them and can warn the user.
// (Codex review 2026/04/17 M1)
for chunk in collected {
if requeue_tx.try_send(chunk).is_err() {
dropped_chunks.fetch_add(1, Ordering::Relaxed);
}
}
eprintln!("[kon-audio] selected microphone: '{name}'");
Ok((
MicrophoneCapture {
stream: Some(stream),
device_name: name.to_string(),
dropped_chunks,
error_rx: Some(err_rx),
},
rx,
))
}
#[allow(clippy::too_many_arguments)]
fn build_input_stream<T>(
device: &cpal::Device,
supported_config: &cpal::SupportedStreamConfig,
sample_rate: u32,
channels: u16,
tx: mpsc::SyncSender<AudioChunk>,
dropped_chunks: Arc<AtomicU64>,
err_tx: mpsc::SyncSender<CaptureRuntimeError>,
device_name: String,
) -> std::result::Result<cpal::Stream, cpal::BuildStreamError>
where
T: Sample + SizedSample,
f32: FromSample<T>,
{
let config: cpal::StreamConfig = supported_config.clone().into();
let err_device_name = device_name.clone();
device.build_input_stream(
&config,
move |data: &[T], _| {
let samples: Vec<f32> = data.iter().copied().map(f32::from_sample).collect();
let chunk = AudioChunk {
samples,
sample_rate,
channels,
};
// try_send fails if the channel is full. Track that explicitly
// rather than swallowing it — Codex review 2026/04/17 caught
// this as a silent-failure risk under sustained load.
if tx.try_send(chunk).is_err() {
dropped_chunks.fetch_add(1, Ordering::Relaxed);
}
},
move |err| {
// Surface stream errors to the live session via err_tx so the
// frontend can show a toast. Also keep the eprintln for ops
// logs. (Codex review 2026/04/17 M2)
eprintln!("[kon-audio] capture error: {err}");
let _ = err_tx.try_send(CaptureRuntimeError {
device_name: err_device_name.clone(),
message: err.to_string(),
});
},
None,
)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn monitor_pattern_detection() {
assert!(is_monitor_name(
"alsa_output.pci-0000_00_1f.3.analog-stereo.monitor"
));
assert!(is_monitor_name("Monitor of Built-in Audio Analog Stereo"));
assert!(is_monitor_name("Some Loopback Device"));
assert!(!is_monitor_name("Blue Yeti USB"));
assert!(!is_monitor_name(
"alsa_input.pci-0000_00_1f.3.analog-stereo"
));
assert!(!is_monitor_name(""));
}
}