Files
Lumotia/crates/transcription/src/whisper_rs_backend.rs
Jake 34fce3cf9e
Some checks failed
check / cargo check (macos-latest) (push) Has been cancelled
check / cargo check (ubuntu-22.04) (push) Has been cancelled
check / cargo check (windows-latest) (push) Has been cancelled
check / svelte build + lint (push) Has been cancelled
feat: OpenWhispr-inspired transcription polish pass
Major quality pass on top of Phase 2. Five substantive changes plus
cross-cutting touches across audio, hotkey, transcription, and Tauri
command layers.

  Transcription quality

  - Long-audio chunking in commands/transcription.rs: Parakeet and large
    file transcription now chunk-and-recompose with overlap trimming, so
    the live-path chunking advantage extends to file-based workflows.
  - Stateful live speech gate in commands/live.rs on top of the earlier
    duplicate-boundary filtering — distinguishes start-of-speech from
    mid-speech and holds state across chunks.

  Auto-learning corrections

  - New crates/ai-formatting/src/correction_learning.rs: extracts user
    text corrections from viewer edits and proposes additions to the
    active profile's vocabulary.
  - src-tauri/src/commands/profiles.rs bridge for frontend-driven
    confirmation of learned terms.
  - src/routes/viewer/+page.svelte hooks the learning path into the
    segment-edit flow so corrections feed profile_terms without a
    separate 'train this profile' UX.

  Transcript profile provenance

  - Migration v8 (crates/storage/src/migrations.rs) adds profile_id to
    transcripts, defaulting to DEFAULT_PROFILE_ID so existing rows stay
    valid.
  - crates/storage/src/database.rs: TranscriptRow + CRUD carry profile_id.
  - src-tauri/src/commands/transcripts.rs: add_transcript accepts and
    persists profile_id.
  - DictationPage.svelte + FilesPage.svelte send activeProfileId on
    capture so learned corrections are attributed to the right profile.

  Cleanup prompt contract

  - crates/ai-formatting/src/llm_client.rs hardened: the CLEANUP_PROMPT
    now specifies concrete do/do-not rules, ready for a real model-backed
    cleanup pass. The llm_client is still a stub — kon-llm remains unwired
    — but the prompt shape is final.

  Cross-cutting polish

  - Minor touches in audio (capture/decode/resample), hotkey (lib/linux/stub),
    core, transcription (concurrency/model_manager/local_engine/whisper_rs),
    and the rest of src-tauri/src/commands/*: error-path tightening, log
    clarity, TS-migration follow-ups (@ts-nocheck additions for incremental
    typing).

Verified locally: npm run check, cargo test -p kon-ai-formatting,
cargo test -p kon-storage, cargo test -p kon --lib commands::live::tests,
cargo check — all green.

Scope boundary: kon-llm crate is still a stub; task extraction remains
rule-based. Bundled local-LLM runtime is the next clean step and is not
in this commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 22:39:08 +01:00

105 lines
3.4 KiB
Rust

//! Direct whisper-rs backend. Owns a WhisperContext; each call builds a
//! fresh WhisperState (state can be reused, but fresh-per-call is simpler
//! and matches the transcribe-rs call style we are replacing).
//!
//! Exists because transcribe-rs does not expose set_initial_prompt; this
//! wrapper is the only path that can pipe per-capture vocabulary context
//! into Whisper.
use std::path::Path;
use whisper_rs::{FullParams, SamplingStrategy, WhisperContext, WhisperContextParameters};
use kon_core::types::{Segment, TranscriptionOptions};
#[derive(Debug, thiserror::Error)]
pub enum WhisperBackendError {
#[error("whisper-rs load failed: {0}")]
Load(String),
#[error("whisper-rs state creation failed: {0}")]
State(String),
#[error("whisper-rs transcribe failed: {0}")]
Transcribe(String),
}
pub struct WhisperRsBackend {
ctx: WhisperContext,
}
impl WhisperRsBackend {
pub fn load(model_path: &Path) -> Result<Self, WhisperBackendError> {
let ctx = WhisperContext::new_with_params(model_path, WhisperContextParameters::default())
.map_err(|e| WhisperBackendError::Load(e.to_string()))?;
Ok(Self { ctx })
}
/// Synchronously transcribe 16 kHz mono f32 PCM.
///
/// `options.initial_prompt` is piped directly to whisper-rs.
pub fn transcribe_sync(
&self,
samples: &[f32],
options: &TranscriptionOptions,
) -> Result<Vec<Segment>, WhisperBackendError> {
tracing::info!(
language = ?options.language,
has_initial_prompt = options.initial_prompt.as_deref().map(|p| !p.is_empty()).unwrap_or(false),
"WhisperRsBackend::transcribe_sync entering"
);
let mut state = self
.ctx
.create_state()
.map_err(|e| WhisperBackendError::State(e.to_string()))?;
let mut params = FullParams::new(SamplingStrategy::Greedy { best_of: 1 });
if let Some(lang) = options.language.as_deref() {
if !lang.is_empty() {
params.set_language(Some(lang));
}
}
if let Some(prompt) = options.initial_prompt.as_deref() {
if !prompt.is_empty() {
params.set_initial_prompt(prompt);
}
}
params.set_n_threads(num_cpus::get() as i32);
params.set_print_special(false);
params.set_print_progress(false);
params.set_print_realtime(false);
state
.full(params, samples)
.map_err(|e| WhisperBackendError::Transcribe(e.to_string()))?;
let n = state.full_n_segments();
let mut out = Vec::with_capacity(n.max(0) as usize);
for i in 0..n {
let Some(seg) = state.get_segment(i) else {
continue;
};
let text = seg
.to_str()
.map_err(|e| WhisperBackendError::Transcribe(e.to_string()))?
.to_string();
// whisper-rs timestamps are centiseconds (10ms units). Convert to seconds (f64).
let start = seg.start_timestamp() as f64 * 0.01;
let end = seg.end_timestamp() as f64 * 0.01;
out.push(Segment { start, end, text });
}
Ok(out)
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn backend_error_displays() {
let e = WhisperBackendError::Load("oops".into());
assert!(e.to_string().contains("oops"));
}
}