jake/Lumotia
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: OpenWhispr-inspired transcription polish pass
Some checks failed
check / cargo check (macos-latest) (push) Has been cancelled
Details
check / cargo check (ubuntu-22.04) (push) Has been cancelled
Details
check / cargo check (windows-latest) (push) Has been cancelled
Details
check / svelte build + lint (push) Has been cancelled
Details

Browse Source 
Major quality pass on top of Phase 2. Five substantive changes plus
cross-cutting touches across audio, hotkey, transcription, and Tauri
command layers.

  Transcription quality

  - Long-audio chunking in commands/transcription.rs: Parakeet and large
    file transcription now chunk-and-recompose with overlap trimming, so
    the live-path chunking advantage extends to file-based workflows.
  - Stateful live speech gate in commands/live.rs on top of the earlier
    duplicate-boundary filtering — distinguishes start-of-speech from
    mid-speech and holds state across chunks.

  Auto-learning corrections

  - New crates/ai-formatting/src/correction_learning.rs: extracts user
    text corrections from viewer edits and proposes additions to the
    active profile's vocabulary.
  - src-tauri/src/commands/profiles.rs bridge for frontend-driven
    confirmation of learned terms.
  - src/routes/viewer/+page.svelte hooks the learning path into the
    segment-edit flow so corrections feed profile_terms without a
    separate 'train this profile' UX.

  Transcript profile provenance

  - Migration v8 (crates/storage/src/migrations.rs) adds profile_id to
    transcripts, defaulting to DEFAULT_PROFILE_ID so existing rows stay
    valid.
  - crates/storage/src/database.rs: TranscriptRow + CRUD carry profile_id.
  - src-tauri/src/commands/transcripts.rs: add_transcript accepts and
    persists profile_id.
  - DictationPage.svelte + FilesPage.svelte send activeProfileId on
    capture so learned corrections are attributed to the right profile.

  Cleanup prompt contract

  - crates/ai-formatting/src/llm_client.rs hardened: the CLEANUP_PROMPT
    now specifies concrete do/do-not rules, ready for a real model-backed
    cleanup pass. The llm_client is still a stub — kon-llm remains unwired
    — but the prompt shape is final.

  Cross-cutting polish

  - Minor touches in audio (capture/decode/resample), hotkey (lib/linux/stub),
    core, transcription (concurrency/model_manager/local_engine/whisper_rs),
    and the rest of src-tauri/src/commands/*: error-path tightening, log
    clarity, TS-migration follow-ups (@ts-nocheck additions for incremental
    typing).

Verified locally: npm run check, cargo test -p kon-ai-formatting,
cargo test -p kon-storage, cargo test -p kon --lib commands::live::tests,
cargo check — all green.

Scope boundary: kon-llm crate is still a stub; task extraction remains
rule-based. Bundled local-LLM runtime is the next clean step and is not
in this commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jake
2026-04-19 22:39:08 +01:00
parent 28acdcfa6d
commit 34fce3cf9e
39 changed files with 1581 additions and 554 deletions
Download Patch File Download Diff File Expand all files Collapse all files

41
src-tauri/src/commands/audio.rs
View File

@@ -67,18 +67,17 @@ pub async fn start_native_capture(
// worst case). Run on a blocking thread so the async runtime stays
// responsive to other Tauri commands. (Codex review 2026/04/17 D2)
let device_name_for_blocking = device_name.clone();
let (capture, rx) = tokio::task::spawn_blocking(move || {
match device_name_for_blocking.as_deref() {
let (capture, rx) =
tokio::task::spawn_blocking(move || match device_name_for_blocking.as_deref() {
Some(name) if !name.is_empty() => MicrophoneCapture::start_with_device(name),
_ => MicrophoneCapture::start(),
}
})
.await
.map_err(|e| format!("audio task join error: {e}"))?
.map_err(|e| {
eprintln!("[native-capture] MicrophoneCapture::start failed: {e}");
e.to_string()
})?;
})
.await
.map_err(|e| format!("audio task join error: {e}"))?
.map_err(|e| {
eprintln!("[native-capture] MicrophoneCapture::start failed: {e}");
e.to_string()
})?;
eprintln!(
"[native-capture] cpal capture started successfully on '{}'",
capture.device_name
@@ -150,7 +149,9 @@ pub async fn start_native_capture(
}
Err(std::sync::mpsc::TryRecvError::Empty) => break,
Err(std::sync::mpsc::TryRecvError::Disconnected) => {
eprintln!("[native-capture] capture stream disconnected; accumulator exiting");
eprintln!(
"[native-capture] capture stream disconnected; accumulator exiting"
);
capture_dead = true;
break;
}
@@ -166,9 +167,12 @@ pub async fn start_native_capture(
all.extend_from_slice(&chunk);
}
let _ = app.emit("native-pcm", serde_json::json!({
"samples": chunk,
}));
let _ = app.emit(
"native-pcm",
serde_json::json!({
"samples": chunk,
}),
);
}
if capture_dead {
@@ -185,9 +189,12 @@ pub async fn start_native_capture(
if let Ok(mut all) = all_samples_clone.lock() {
all.extend_from_slice(&pcm_buffer);
}
let _ = app.emit("native-pcm", serde_json::json!({
"samples": pcm_buffer,
}));
let _ = app.emit(
"native-pcm",
serde_json::json!({
"samples": pcm_buffer,
}),
);
}
// Drop the capture to stop the cpal stream

3
src-tauri/src/commands/clipboard.rs
View File

@@ -3,8 +3,7 @@ use arboard::Clipboard;
/// Copy text to the system clipboard via arboard.
#[tauri::command]
pub fn copy_to_clipboard(text: String) -> Result<(), String> {
let mut clipboard =
Clipboard::new().map_err(|e| format!("Clipboard init failed: {e}"))?;
let mut clipboard = Clipboard::new().map_err(|e| format!("Clipboard init failed: {e}"))?;
clipboard
.set_text(&text)
.map_err(|e| format!("Clipboard write failed: {e}"))?;

22
src-tauri/src/commands/diagnostics.rs
View File

@@ -60,9 +60,7 @@ pub fn install_panic_hook() {
RUST_BACKTRACE: {bt}\n",
ver = KON_VERSION,
ts = ts,
thread = std::thread::current()
.name()
.unwrap_or("<unnamed>"),
thread = std::thread::current().name().unwrap_or("<unnamed>"),
info = info,
os = std::env::consts::OS,
arch = std::env::consts::ARCH,
@@ -98,7 +96,11 @@ pub async fn log_frontend_error(
log_error(
&state.db,
if context.is_empty() { "frontend" } else { &context },
if context.is_empty() {
"frontend"
} else {
&context
},
Some("FRONTEND_ERROR"),
&message,
metadata.as_deref(),
@@ -245,8 +247,10 @@ pub async fn generate_diagnostic_report(
.unwrap_or(0);
out.push_str(&format!("- Generated: unix `{}`\n", now));
out.push_str("\n");
out.push_str("> This report is local-only until you choose to share it. \
Review the contents below before sending to anyone.\n\n");
out.push_str(
"> This report is local-only until you choose to share it. \
Review the contents below before sending to anyone.\n\n",
);
if opts.include_settings {
out.push_str("## Settings (sanitised)\n\n");
@@ -325,8 +329,10 @@ pub async fn generate_diagnostic_report(
}
out.push_str("---\n\n");
out.push_str("Generated by Kon. To share, copy the entire markdown above \
and paste it into an email or issue. Email: jake@corbel.consulting.\n");
out.push_str(
"Generated by Kon. To share, copy the entire markdown above \
and paste it into an email or issue. Email: jake@corbel.consulting.\n",
);
Ok(out)
}

4
src-tauri/src/commands/hotkey.rs
View File

@@ -96,9 +96,7 @@ pub async fn update_evdev_hotkey(
/// Stop the evdev hotkey listener.
#[tauri::command]
pub async fn stop_evdev_hotkey(
state: tauri::State<'_, HotkeyState>,
) -> Result<(), String> {
pub async fn stop_evdev_hotkey(state: tauri::State<'_, HotkeyState>) -> Result<(), String> {
let mut guard = state.listener.lock().await;
if let Some(listener) = guard.take() {
listener.stop().await;

559
src-tauri/src/commands/live.rs
View File

@@ -1,5 +1,6 @@
#![allow(clippy::too_many_arguments)]
use std::collections::HashMap;
use std::sync::{
atomic::{AtomicBool, AtomicU64, Ordering},
Arc, Mutex,
@@ -13,9 +14,7 @@ use tauri::ipc::Channel;
use crate::commands::audio::persist_audio_samples;
use crate::commands::models::{default_model_id_for_engine, ensure_model_loaded};
use crate::AppState;
use kon_ai_formatting::{
post_process_segments, FormatMode, PostProcessOptions,
};
use kon_ai_formatting::{post_process_segments, FormatMode, PostProcessOptions};
use kon_audio::{MicrophoneCapture, StreamingResampler};
use kon_core::constants::WHISPER_SAMPLE_RATE;
use kon_core::types::{AudioSamples, Segment, TranscriptionOptions};
@@ -27,9 +26,30 @@ const FINAL_CHUNK_MIN_SAMPLES: usize = 4_000; // 0.25s
const MAX_PENDING_SAMPLES: usize = CHUNK_SAMPLES;
const SPEECH_FRAME_SAMPLES: usize = 800; // 50ms
const MIN_SPEECH_FRAMES: usize = 1; // any plausible speech-like frame
const RMS_SPEECH_THRESHOLD: f32 = 0.001;
const PEAK_SPEECH_THRESHOLD: f32 = 0.004;
const SILENCE_RMS_THRESHOLD: f32 = 0.001;
const SPEECH_WINDOW_RMS_THRESHOLD: f32 = 0.0014;
const SPEECH_WINDOW_PEAK_THRESHOLD: f32 = 0.004;
const STRONG_SPEECH_RMS_THRESHOLD: f32 = 0.003;
const STRONG_SPEECH_PEAK_THRESHOLD: f32 = 0.012;
const FLATLINE_PEAK_THRESHOLD: f32 = 0.0005;
const DUPLICATE_TRANSCRIPT_WINDOW_SECS: f64 = 6.0;
const DUPLICATE_TRANSCRIPT_MERGE_LIMIT: usize = 3;
const DUPLICATE_HISTORY_RETENTION_SECS: f64 = 8.0;
const DUPLICATE_CHECK_LEADING_SECS: f64 = 1.5;
const TOKEN_COVERAGE_THRESHOLD: f64 = 0.6;
const TOKEN_SEQUENCE_THRESHOLD: f64 = 0.6;
const MIN_TOKENS_FOR_OVERLAP: usize = 3;
const MEANINGFUL_TOKEN_COVERAGE_THRESHOLD: f64 = 0.55;
const MEANINGFUL_TOKEN_SEQUENCE_THRESHOLD: f64 = 0.55;
const MIN_MEANINGFUL_TOKENS_FOR_OVERLAP: usize = 4;
const LOW_SIGNAL_TOKENS: &[&str] = &[
"a", "an", "and", "are", "as", "at", "be", "been", "being", "but", "by", "d", "did", "do",
"does", "for", "from", "had", "has", "have", "he", "her", "here", "his", "how", "i", "if",
"in", "is", "it", "ll", "m", "me", "my", "of", "on", "or", "our", "out", "re", "s", "she",
"so", "t", "that", "the", "their", "them", "there", "these", "they", "this", "those", "to",
"ve", "was", "we", "well", "were", "what", "when", "where", "which", "who", "why", "with",
"without", "you", "your",
];
#[derive(Default)]
pub struct LiveTranscriptionState {
@@ -131,6 +151,34 @@ struct InferenceTask {
rx: std::sync::mpsc::Receiver<Result<kon_transcription::TimedTranscript, String>>,
}
#[derive(Debug, Clone)]
struct RecentTranscriptSegment {
start_secs: f64,
end_secs: f64,
text: String,
}
#[derive(Debug, Clone, Copy, Default)]
struct SpeechGateState {
peak_rms: f32,
peak_amplitude: f32,
window_count: usize,
speech_window_count: usize,
consecutive_speech_windows: usize,
max_consecutive_speech_windows: usize,
}
#[derive(Debug, Clone, Copy, PartialEq)]
struct SpeechGateDecision {
skip: bool,
reason: &'static str,
peak_rms: f32,
peak_amplitude: f32,
window_count: usize,
speech_window_count: usize,
max_consecutive_speech_windows: usize,
}
#[tauri::command]
pub async fn start_live_transcription_session(
state: tauri::State<'_, AppState>,
@@ -183,10 +231,7 @@ pub async fn start_live_transcription_session(
.unwrap_or_else(|| default_model_id_for_engine(&config.engine).to_string());
eprintln!(
"[live] starting session: engine={}, model={}, language={:?}, save_audio={}",
config.engine,
model_id,
config.language,
config.save_audio
config.engine, model_id, config.language, config.save_audio
);
ensure_model_loaded(&state, &config.engine, &model_id).await?;
@@ -250,10 +295,7 @@ pub async fn stop_live_transcription_session(
.map_err(|e| format!("Live session task failed: {e}"))??;
let audio_path = if let Some(samples) = summary.audio_samples {
Some(
persist_audio_samples(&app, samples, running.output_folder.clone())
.await?,
)
Some(persist_audio_samples(&app, samples, running.output_folder.clone()).await?)
} else {
None
};
@@ -273,10 +315,7 @@ pub async fn stop_live_transcription_session(
Ok(response)
}
fn pick_engine(
state: &AppState,
engine: &str,
) -> Result<Arc<LocalEngine>, String> {
fn pick_engine(state: &AppState, engine: &str) -> Result<Arc<LocalEngine>, String> {
match engine {
"whisper" => Ok(state.whisper_engine.clone()),
"parakeet" => Ok(state.parakeet_engine.clone()),
@@ -317,12 +356,14 @@ fn run_live_session(
let mut chunk_id: u32 = 0;
let mut inflight: Option<InferenceTask> = None;
let mut resampler_flushed = false;
let mut recent_segments: Vec<RecentTranscriptSegment> = Vec::new();
loop {
if let Some(_done) = poll_inference(
&mut inflight,
session_id,
&config,
&mut recent_segments,
&dictionary_terms,
&result_channel,
&status_channel,
@@ -358,18 +399,12 @@ fn run_live_session(
}
};
let resampled =
resampler.push_samples(&mono).map_err(|e| e.to_string())?;
append_resampled_audio(
&mut capture_buffer,
&mut kept_audio,
&resampled,
);
let resampled = resampler.push_samples(&mono).map_err(|e| e.to_string())?;
append_resampled_audio(&mut capture_buffer, &mut kept_audio, &resampled);
}
Err(std::sync::mpsc::RecvTimeoutError::Timeout) => {}
Err(std::sync::mpsc::RecvTimeoutError::Disconnected) => {
let message =
"Microphone capture disconnected unexpectedly".to_string();
let message = "Microphone capture disconnected unexpectedly".to_string();
let _ = status_channel.send(LiveStatusMessage::Error {
session_id,
message: message.clone(),
@@ -426,6 +461,7 @@ fn run_live_session(
&mut inflight,
session_id,
&config,
&mut recent_segments,
&dictionary_terms,
&result_channel,
&status_channel,
@@ -485,17 +521,33 @@ fn maybe_dispatch_chunk(
&capture_buffer[..target_len]
};
if !has_enough_speech(speech_window) {
let skipped_ms =
(target_len as u64 * 1000) / WHISPER_SAMPLE_RATE as u64;
let speech_gate = evaluate_speech_gate(speech_window);
if speech_gate.skip {
let skipped_ms = (target_len as u64 * 1000) / WHISPER_SAMPLE_RATE as u64;
let gate_reason = match speech_gate.reason {
"silence" => "near-silence",
"insufficient_speech" => "insufficient speech energy",
other => other,
};
eprintln!(
"[live] session {session_id}: skipped {skipped_ms}ms chunk as near-silence"
"[live] session {session_id}: skipped {skipped_ms}ms chunk as {gate_reason} \
(peak_rms={:.6}, peak={:.6}, speech_windows={}/{}, max_consecutive={})",
speech_gate.peak_rms,
speech_gate.peak_amplitude,
speech_gate.speech_window_count,
speech_gate.window_count,
speech_gate.max_consecutive_speech_windows,
);
let _ = status_channel.send(LiveStatusMessage::Warning {
session_id,
message: format!(
"Skipped {skipped_ms}ms of near-silent audio. If this keeps happening, try a louder mic level or move closer to the microphone."
),
message: match speech_gate.reason {
"silence" => format!(
"Skipped {skipped_ms}ms of near-silent audio. If this keeps happening, try a louder mic level or move closer to the microphone."
),
_ => format!(
"Skipped {skipped_ms}ms of low-confidence audio. If this keeps happening, try a louder mic level or reduce background noise."
),
},
});
capture_buffer.drain(..target_len);
*buffer_start_sample = buffer_start_sample.saturating_add(target_len as u64);
@@ -553,6 +605,7 @@ fn poll_inference(
inflight: &mut Option<InferenceTask>,
session_id: u64,
config: &StartLiveTranscriptionConfig,
recent_segments: &mut Vec<RecentTranscriptSegment>,
dictionary_terms: &[String],
result_channel: &Channel<LiveResultMessage>,
status_channel: &Channel<LiveStatusMessage>,
@@ -563,8 +616,7 @@ fn poll_inference(
match task.rx.try_recv() {
Ok(Ok(timed)) => {
let mut segments: Vec<Segment> =
timed.transcript.segments().to_vec();
let mut segments: Vec<Segment> = timed.transcript.segments().to_vec();
trim_overlap_segments(&mut segments, task.trim_before_secs);
post_process_segments(
&mut segments,
@@ -576,25 +628,37 @@ fn poll_inference(
dictionary_terms: dictionary_terms.to_vec(),
},
);
let chunk_start_secs = task.chunk_start_sample as f64 / WHISPER_SAMPLE_RATE as f64;
let skipped_duplicates = filter_duplicate_boundary_segments(
&mut segments,
chunk_start_secs,
recent_segments,
);
let segment_count = segments.len();
let delivered_segments = segments.clone();
result_channel
.send(LiveResultMessage {
session_id,
chunk_id: task.chunk_id,
chunk_start_secs: task.chunk_start_sample as f64
/ WHISPER_SAMPLE_RATE as f64,
chunk_start_secs,
duration: task.duration_secs,
language: timed.transcript.language().to_string(),
inference_ms: timed.inference_ms,
segments,
})
.map_err(|e| e.to_string())?;
remember_recent_segments(recent_segments, &delivered_segments, chunk_start_secs);
eprintln!(
"[live] session {session_id}: delivered chunk {} with {} segments in {}ms",
"[live] session {session_id}: delivered chunk {} with {} segments in {}ms{}",
task.chunk_id,
segment_count,
timed.inference_ms
timed.inference_ms,
if skipped_duplicates > 0 {
format!(" (skipped {skipped_duplicates} duplicate boundary segment(s))")
} else {
String::new()
}
);
*inflight = None;
@@ -636,34 +700,314 @@ fn trim_overlap_segments(segments: &mut Vec<Segment>, trim_before_secs: f64) {
}
}
fn has_enough_speech(samples: &[f32]) -> bool {
if samples.is_empty() {
fn filter_duplicate_boundary_segments(
segments: &mut Vec<Segment>,
chunk_start_secs: f64,
recent_segments: &[RecentTranscriptSegment],
) -> usize {
if recent_segments.is_empty() {
return 0;
}
let mut skipped = 0usize;
segments.retain(|segment| {
if segment.start > DUPLICATE_CHECK_LEADING_SECS {
return true;
}
let absolute_start = chunk_start_secs + segment.start;
let candidates = build_nearby_transcript_candidates(recent_segments, absolute_start);
if candidates.is_empty() {
return true;
}
let duplicate = candidates.iter().any(|candidate| {
transcripts_overlap(&segment.text, candidate)
|| transcripts_loosely_overlap(&segment.text, candidate)
});
if duplicate {
skipped += 1;
return false;
}
true
});
skipped
}
fn remember_recent_segments(
recent_segments: &mut Vec<RecentTranscriptSegment>,
segments: &[Segment],
chunk_start_secs: f64,
) {
if segments.is_empty() {
return;
}
for segment in segments {
let text = segment.text.trim();
if text.is_empty() {
continue;
}
recent_segments.push(RecentTranscriptSegment {
start_secs: chunk_start_secs + segment.start,
end_secs: chunk_start_secs + segment.end,
text: text.to_string(),
});
}
let cutoff = recent_segments
.last()
.map(|segment| segment.end_secs - DUPLICATE_HISTORY_RETENTION_SECS)
.unwrap_or(0.0);
recent_segments.retain(|segment| segment.end_secs >= cutoff);
}
fn build_nearby_transcript_candidates(
recent_segments: &[RecentTranscriptSegment],
timestamp_secs: f64,
) -> Vec<String> {
let mut nearby: Vec<&RecentTranscriptSegment> = recent_segments
.iter()
.filter(|segment| {
!segment.text.trim().is_empty()
&& (segment.end_secs - timestamp_secs).abs() <= DUPLICATE_TRANSCRIPT_WINDOW_SECS
})
.collect();
nearby.sort_by(|left, right| {
left.start_secs
.partial_cmp(&right.start_secs)
.unwrap_or(std::cmp::Ordering::Equal)
});
let mut texts: Vec<String> = Vec::new();
for start in 0..nearby.len() {
let mut merged = String::new();
for end in start..nearby.len().min(start + DUPLICATE_TRANSCRIPT_MERGE_LIMIT) {
if !merged.is_empty() {
merged.push(' ');
}
merged.push_str(nearby[end].text.trim());
if !texts.iter().any(|existing| existing == &merged) {
texts.push(merged.clone());
}
}
}
texts
}
fn normalize_transcript_text(text: &str) -> String {
let mut normalized = String::with_capacity(text.len());
for ch in text.chars() {
if ch.is_alphanumeric() {
for lower in ch.to_lowercase() {
normalized.push(lower);
}
} else {
normalized.push(' ');
}
}
normalized.split_whitespace().collect::<Vec<_>>().join(" ")
}
fn count_common_tokens<'a>(a: &[&'a str], b: &[&'a str]) -> usize {
let mut counts: HashMap<&'a str, usize> = HashMap::new();
for token in a {
*counts.entry(*token).or_insert(0) += 1;
}
let mut common = 0usize;
for token in b {
if let Some(remaining) = counts.get_mut(*token) {
if *remaining > 0 {
*remaining -= 1;
common += 1;
}
}
}
common
}
fn longest_common_token_subsequence(a: &[&str], b: &[&str]) -> usize {
let mut prev = vec![0usize; b.len() + 1];
let mut curr = vec![0usize; b.len() + 1];
for token_a in a {
for (j, token_b) in b.iter().enumerate() {
curr[j + 1] = if token_a == token_b {
prev[j] + 1
} else {
prev[j + 1].max(curr[j])
};
}
prev.clone_from(&curr);
curr.fill(0);
}
prev[b.len()]
}
fn is_low_signal_token(token: &str) -> bool {
LOW_SIGNAL_TOKENS
.iter()
.any(|low_signal| *low_signal == token)
}
fn meaningful_tokens<'a>(text: &'a str) -> Vec<&'a str> {
text.split_whitespace()
.filter(|token| !token.is_empty() && token.len() > 1 && !is_low_signal_token(token))
.collect()
}
fn transcripts_overlap(a: &str, b: &str) -> bool {
let normalized_a = normalize_transcript_text(a);
let normalized_b = normalize_transcript_text(b);
if normalized_a.is_empty() || normalized_b.is_empty() {
return false;
}
if normalized_a == normalized_b
|| normalized_a.contains(&normalized_b)
|| normalized_b.contains(&normalized_a)
{
return true;
}
let tokens_a: Vec<&str> = normalized_a.split_whitespace().collect();
let tokens_b: Vec<&str> = normalized_b.split_whitespace().collect();
let shorter = tokens_a.len().min(tokens_b.len());
if shorter < MIN_TOKENS_FOR_OVERLAP {
return false;
}
let common = count_common_tokens(&tokens_a, &tokens_b);
if common as f64 / shorter as f64 >= TOKEN_COVERAGE_THRESHOLD {
return true;
}
let sequence = longest_common_token_subsequence(&tokens_a, &tokens_b);
sequence as f64 / shorter as f64 >= TOKEN_SEQUENCE_THRESHOLD
}
fn transcripts_loosely_overlap(a: &str, b: &str) -> bool {
let normalized_a = normalize_transcript_text(a);
let normalized_b = normalize_transcript_text(b);
if normalized_a.is_empty() || normalized_b.is_empty() {
return false;
}
if normalized_a == normalized_b
|| normalized_a.contains(&normalized_b)
|| normalized_b.contains(&normalized_a)
{
return true;
}
let tokens_a = meaningful_tokens(&normalized_a);
let tokens_b = meaningful_tokens(&normalized_b);
let shorter = tokens_a.len().min(tokens_b.len());
if shorter < MIN_MEANINGFUL_TOKENS_FOR_OVERLAP {
return false;
}
let common = count_common_tokens(&tokens_a, &tokens_b);
if common as f64 / shorter as f64 >= MEANINGFUL_TOKEN_COVERAGE_THRESHOLD {
return true;
}
let sequence = longest_common_token_subsequence(&tokens_a, &tokens_b);
sequence as f64 / shorter as f64 >= MEANINGFUL_TOKEN_SEQUENCE_THRESHOLD
}
fn record_speech_window(state: &mut SpeechGateState, rms: f32, peak: f32) {
state.window_count += 1;
state.peak_rms = state.peak_rms.max(rms);
state.peak_amplitude = state.peak_amplitude.max(peak);
let is_speech_window =
rms >= SPEECH_WINDOW_RMS_THRESHOLD && peak >= SPEECH_WINDOW_PEAK_THRESHOLD;
if !is_speech_window {
state.consecutive_speech_windows = 0;
return;
}
state.speech_window_count += 1;
state.consecutive_speech_windows += 1;
state.max_consecutive_speech_windows = state
.max_consecutive_speech_windows
.max(state.consecutive_speech_windows);
}
fn speech_gate_decision(state: SpeechGateState, chunk_peak: f32) -> SpeechGateDecision {
if state.window_count == 0 {
return SpeechGateDecision {
skip: false,
reason: "unavailable",
peak_rms: state.peak_rms,
peak_amplitude: state.peak_amplitude,
window_count: state.window_count,
speech_window_count: state.speech_window_count,
max_consecutive_speech_windows: state.max_consecutive_speech_windows,
};
}
let reason = if chunk_peak < FLATLINE_PEAK_THRESHOLD || state.peak_rms < SILENCE_RMS_THRESHOLD {
Some("silence")
} else if state.speech_window_count < MIN_SPEECH_FRAMES
&& state.peak_rms < STRONG_SPEECH_RMS_THRESHOLD
&& state.peak_amplitude < STRONG_SPEECH_PEAK_THRESHOLD
{
Some("insufficient_speech")
} else {
None
};
SpeechGateDecision {
skip: reason.is_some(),
reason: reason.unwrap_or("speech_detected"),
peak_rms: state.peak_rms,
peak_amplitude: state.peak_amplitude,
window_count: state.window_count,
speech_window_count: state.speech_window_count,
max_consecutive_speech_windows: state.max_consecutive_speech_windows,
}
}
fn evaluate_speech_gate(samples: &[f32]) -> SpeechGateDecision {
if samples.is_empty() {
return SpeechGateDecision {
skip: true,
reason: "silence",
peak_rms: 0.0,
peak_amplitude: 0.0,
window_count: 0,
speech_window_count: 0,
max_consecutive_speech_windows: 0,
};
}
let chunk_peak = samples
.iter()
.map(|sample| sample.abs())
.fold(0.0_f32, f32::max);
if chunk_peak < FLATLINE_PEAK_THRESHOLD {
return false;
}
let mut speech_frames = 0usize;
let mut state = SpeechGateState::default();
for frame in samples.chunks(SPEECH_FRAME_SAMPLES) {
let len = frame.len().max(1) as f32;
let rms = (frame.iter().map(|sample| sample * sample).sum::<f32>() / len)
.sqrt();
let rms = (frame.iter().map(|sample| sample * sample).sum::<f32>() / len).sqrt();
let peak = frame
.iter()
.map(|sample| sample.abs())
.fold(0.0_f32, f32::max);
if rms >= RMS_SPEECH_THRESHOLD || peak >= PEAK_SPEECH_THRESHOLD {
speech_frames += 1;
}
record_speech_window(&mut state, rms, peak);
}
speech_frames >= MIN_SPEECH_FRAMES
speech_gate_decision(state, chunk_peak)
}
fn downmix_chunk(samples: Vec<f32>, channels: usize) -> Vec<f32> {
@@ -676,3 +1020,114 @@ fn downmix_chunk(samples: Vec<f32>, channels: usize) -> Vec<f32> {
.map(|frame| frame.iter().sum::<f32>() / channels as f32)
.collect()
}
#[cfg(test)]
mod tests {
use super::*;
fn segment(start: f64, end: f64, text: &str) -> Segment {
Segment {
start,
end,
text: text.to_string(),
}
}
#[test]
fn transcripts_overlap_detects_boundary_repeat() {
assert!(transcripts_overlap(
"I need to go to the shops tomorrow",
"to go to the shops tomorrow"
));
}
#[test]
fn loose_overlap_ignores_low_signal_only_match() {
assert!(!transcripts_loosely_overlap(
"I think we should do that soon",
"we should maybe do it soon"
));
}
#[test]
fn duplicate_boundary_filter_skips_repeated_opening_segment() {
let recent_segments = vec![RecentTranscriptSegment {
start_secs: 10.0,
end_secs: 12.0,
text: "I need to go to the shops tomorrow".to_string(),
}];
let mut segments = vec![
segment(0.2, 1.0, "Need to go to the shops tomorrow"),
segment(1.8, 2.4, "While I am there I need some cheese"),
];
let skipped = filter_duplicate_boundary_segments(&mut segments, 11.8, &recent_segments);
assert_eq!(skipped, 1);
assert_eq!(segments.len(), 1);
assert_eq!(segments[0].text, "While I am there I need some cheese");
}
#[test]
fn remember_recent_segments_prunes_old_history() {
let mut recent_segments = vec![RecentTranscriptSegment {
start_secs: 0.0,
end_secs: 1.0,
text: "old text".to_string(),
}];
remember_recent_segments(
&mut recent_segments,
&[segment(0.0, 0.8, "new text")],
DUPLICATE_HISTORY_RETENTION_SECS + 1.0,
);
assert_eq!(recent_segments.len(), 1);
assert_eq!(recent_segments[0].text, "new text");
}
#[test]
fn speech_gate_treats_near_silence_as_skippable() {
let samples = vec![0.0004_f32, 0.0002, 0.0003, 0.0001]
.into_iter()
.cycle()
.take(SPEECH_FRAME_SAMPLES * 3)
.collect::<Vec<_>>();
let decision = evaluate_speech_gate(&samples);
assert!(decision.skip);
assert_eq!(decision.reason, "silence");
}
#[test]
fn speech_gate_rejects_isolated_noise_without_speech_windows() {
let mut samples = Vec::new();
for i in 0..(SPEECH_FRAME_SAMPLES * 3) {
let sample = if i % SPEECH_FRAME_SAMPLES == 0 {
0.010
} else {
0.0011
};
samples.push(sample);
}
let decision = evaluate_speech_gate(&samples);
assert!(decision.skip);
assert_eq!(decision.reason, "insufficient_speech");
assert_eq!(decision.speech_window_count, 0);
}
#[test]
fn speech_gate_allows_sustained_speech_like_audio() {
let samples = vec![0.014_f32; SPEECH_FRAME_SAMPLES * 3];
let decision = evaluate_speech_gate(&samples);
assert!(!decision.skip);
assert_eq!(decision.reason, "speech_detected");
assert_eq!(decision.speech_window_count, 3);
assert_eq!(decision.max_consecutive_speech_windows, 3);
}
}

2
src-tauri/src/commands/mod.rs
View File

@@ -6,8 +6,8 @@ pub mod hotkey;
pub mod live;
pub mod models;
pub mod profiles;
pub mod transcription;
pub mod tasks;
pub mod transcription;
pub mod transcripts;
pub mod update;
pub mod windows;

41
src-tauri/src/commands/models.rs
View File

@@ -4,9 +4,7 @@ use serde::Serialize;
use tauri::Emitter;
use crate::AppState;
use kon_core::model_registry::{
self, Engine, LanguageSupport, ModelEntry,
};
use kon_core::model_registry::{self, Engine, LanguageSupport, ModelEntry};
use kon_core::types::ModelId;
use kon_transcription::model_manager;
use kon_transcription::{load_parakeet, load_whisper, LocalEngine};
@@ -29,10 +27,7 @@ fn parakeet_model_id(name: &str) -> ModelId {
}
}
fn engine_for_name(
state: &AppState,
engine_name: &str,
) -> Result<Arc<LocalEngine>, String> {
fn engine_for_name(state: &AppState, engine_name: &str) -> Result<Arc<LocalEngine>, String> {
match engine_name {
"whisper" => Ok(state.whisper_engine.clone()),
"parakeet" => Ok(state.parakeet_engine.clone()),
@@ -40,9 +35,7 @@ fn engine_for_name(
}
}
fn language_support_info(
language_support: LanguageSupport,
) -> LanguageSupportInfo {
fn language_support_info(language_support: LanguageSupport) -> LanguageSupportInfo {
match language_support {
LanguageSupport::EnglishOnly => LanguageSupportInfo {
kind: "english-only".into(),
@@ -72,9 +65,11 @@ fn model_capability(
}
}
pub fn load_model_from_disk(model_id: &ModelId) -> Result<kon_transcription::SpeechBackend, String> {
let entry = model_registry::find_model(model_id)
.ok_or_else(|| format!("Unknown model: {model_id}"))?;
pub fn load_model_from_disk(
model_id: &ModelId,
) -> Result<kon_transcription::SpeechBackend, String> {
let entry =
model_registry::find_model(model_id).ok_or_else(|| format!("Unknown model: {model_id}"))?;
match entry.engine {
Engine::Whisper => {
@@ -252,8 +247,7 @@ pub fn get_runtime_capabilities(
engines: vec![
EngineRuntimeCapabilities {
id: "whisper".into(),
default_model_id: default_model_id_for_engine("whisper")
.to_string(),
default_model_id: default_model_id_for_engine("whisper").to_string(),
loaded_model_id: whisper
.loaded_model_id()
.map(|model_id| model_id.to_string()),
@@ -262,8 +256,7 @@ pub fn get_runtime_capabilities(
},
EngineRuntimeCapabilities {
id: "parakeet".into(),
default_model_id: default_model_id_for_engine("parakeet")
.to_string(),
default_model_id: default_model_id_for_engine("parakeet").to_string(),
loaded_model_id: parakeet
.loaded_model_id()
.map(|model_id| model_id.to_string()),
@@ -277,10 +270,7 @@ pub fn get_runtime_capabilities(
// --- Whisper model commands ---
#[tauri::command]
pub async fn download_model(
app: tauri::AppHandle,
size: String,
) -> Result<String, String> {
pub async fn download_model(app: tauri::AppHandle, size: String) -> Result<String, String> {
let id = whisper_model_id(&size);
let app_clone = app.clone();
model_manager::download(&id, move |progress| {
@@ -314,10 +304,7 @@ pub fn list_models() -> Result<Vec<String>, String> {
}
#[tauri::command]
pub async fn load_model(
state: tauri::State<'_, AppState>,
size: String,
) -> Result<String, String> {
pub async fn load_model(state: tauri::State<'_, AppState>, size: String) -> Result<String, String> {
let id = whisper_model_id(&size);
ensure_model_loaded(&state, "whisper", id.as_str()).await?;
Ok(format!("Model {} loaded", size))
@@ -375,8 +362,6 @@ pub async fn load_parakeet_model(
}
#[tauri::command]
pub fn check_parakeet_engine(
state: tauri::State<'_, AppState>,
) -> Result<bool, String> {
pub fn check_parakeet_engine(state: tauri::State<'_, AppState>) -> Result<bool, String> {
Ok(state.parakeet_engine.is_loaded())
}

29
src-tauri/src/commands/profiles.rs
View File

@@ -11,6 +11,7 @@
use serde::Serialize;
use kon_ai_formatting::extract_corrections;
use kon_storage::{
add_profile_term as db_add_profile_term, create_profile as db_create_profile,
delete_profile as db_delete_profile, delete_profile_term as db_delete_profile_term,
@@ -21,6 +22,8 @@ use kon_storage::{
use crate::AppState;
const AUTO_LEARNED_NOTE: &str = "Auto-learned from transcript edit";
/// Frontend-facing profile shape. Matches the object the Svelte profile
/// picker + editor will consume.
#[derive(Debug, Clone, Serialize)]
@@ -145,6 +148,32 @@ pub async fn add_profile_term_cmd(
.map_err(|e| e.to_string())
}
#[tauri::command]
pub async fn learn_profile_terms_from_edit_cmd(
state: tauri::State<'_, AppState>,
profile_id: String,
original_text: String,
edited_text: String,
) -> Result<Vec<ProfileTermDto>, String> {
let existing_terms: Vec<String> = db_list_profile_terms(&state.db, &profile_id)
.await
.map_err(|e| e.to_string())?
.into_iter()
.map(|row| row.term)
.collect();
let corrections = extract_corrections(&original_text, &edited_text, &existing_terms);
let mut learned = Vec::new();
for term in corrections {
let row = db_add_profile_term(&state.db, &profile_id, &term, AUTO_LEARNED_NOTE)
.await
.map_err(|e| e.to_string())?;
learned.push(ProfileTermDto::from(row));
}
Ok(learned)
}
#[tauri::command]
pub async fn delete_profile_term_cmd(
state: tauri::State<'_, AppState>,

31
src-tauri/src/commands/tasks.rs
View File

@@ -7,17 +7,11 @@ use serde::{Deserialize, Serialize};
use uuid::Uuid;
use kon_storage::{
complete_subtask_and_check_parent as db_complete_subtask,
complete_task as db_complete_task,
delete_task as db_delete_task,
get_task_by_id as db_get_task,
insert_subtask as db_insert_subtask,
insert_task as db_insert_task,
list_subtasks as db_list_subtasks,
list_tasks as db_list_tasks,
uncomplete_task as db_uncomplete_task,
update_task as db_update_task,
TaskRow,
complete_subtask_and_check_parent as db_complete_subtask, complete_task as db_complete_task,
delete_task as db_delete_task, get_task_by_id as db_get_task,
insert_subtask as db_insert_subtask, insert_task as db_insert_task,
list_subtasks as db_list_subtasks, list_tasks as db_list_tasks,
uncomplete_task as db_uncomplete_task, update_task as db_update_task, TaskRow,
};
use crate::AppState;
@@ -138,9 +132,7 @@ pub async fn update_task_cmd(
}
#[tauri::command]
pub async fn list_tasks_cmd(
state: tauri::State<'_, AppState>,
) -> Result<Vec<TaskDto>, String> {
pub async fn list_tasks_cmd(state: tauri::State<'_, AppState>) -> Result<Vec<TaskDto>, String> {
db_list_tasks(&state.db)
.await
.map(|rows| rows.into_iter().map(TaskDto::from).collect())
@@ -158,10 +150,7 @@ pub async fn complete_task_cmd(
}
#[tauri::command]
pub async fn delete_task_cmd(
state: tauri::State<'_, AppState>,
id: String,
) -> Result<(), String> {
pub async fn delete_task_cmd(state: tauri::State<'_, AppState>, id: String) -> Result<(), String> {
db_delete_task(&state.db, &id)
.await
.map_err(|e| e.to_string())
@@ -195,7 +184,10 @@ pub async fn decompose_and_store(
db_insert_subtask(&state.db, &id, &text, &parent_task_id)
.await
.map_err(|e| e.to_string())?;
if let Some(row) = db_get_task(&state.db, &id).await.map_err(|e| e.to_string())? {
if let Some(row) = db_get_task(&state.db, &id)
.await
.map_err(|e| e.to_string())?
{
created.push(TaskDto::from(row));
}
}
@@ -223,4 +215,3 @@ pub async fn complete_subtask_cmd(
.await
.map_err(|e| e.to_string())
}

154
src-tauri/src/commands/transcription.rs
View File

@@ -10,7 +10,20 @@ use tauri::Emitter;
use crate::commands::models::{default_model_id_for_engine, ensure_model_loaded};
use crate::AppState;
use kon_ai_formatting::{post_process_segments, FormatMode, PostProcessOptions};
use kon_core::types::{Segment, TranscriptionOptions};
use kon_core::constants::WHISPER_SAMPLE_RATE;
use kon_core::types::{AudioSamples, Segment, Transcript, TranscriptionOptions};
const PARAKEET_CHUNK_THRESHOLD_SECS: usize = 18;
const PARAKEET_CHUNK_SECS: usize = 15;
const PARAKEET_CHUNK_OVERLAP_SECS: usize = 1;
const FILE_CHUNK_THRESHOLD_SECS: usize = 8 * 60;
const FILE_CHUNK_SECS: usize = 3 * 60;
const FILE_CHUNK_OVERLAP_SECS: usize = 2;
struct ChunkingStrategy {
chunk_samples: usize,
overlap_samples: usize,
}
fn pick_engine(
state: &AppState,
@@ -23,6 +36,104 @@ fn pick_engine(
}
}
fn pick_chunking_strategy(engine_name: &str, sample_count: usize) -> Option<ChunkingStrategy> {
let samples_per_second = WHISPER_SAMPLE_RATE as usize;
match engine_name {
"parakeet" if sample_count > PARAKEET_CHUNK_THRESHOLD_SECS * samples_per_second => {
Some(ChunkingStrategy {
chunk_samples: PARAKEET_CHUNK_SECS * samples_per_second,
overlap_samples: PARAKEET_CHUNK_OVERLAP_SECS * samples_per_second,
})
}
_ if sample_count > FILE_CHUNK_THRESHOLD_SECS * samples_per_second => {
Some(ChunkingStrategy {
chunk_samples: FILE_CHUNK_SECS * samples_per_second,
overlap_samples: FILE_CHUNK_OVERLAP_SECS * samples_per_second,
})
}
_ => None,
}
}
fn trim_overlap_segments(segments: &mut Vec<Segment>, trim_before_secs: f64) {
if trim_before_secs <= 0.0 {
return;
}
segments.retain(|segment| segment.end > trim_before_secs);
for segment in segments.iter_mut() {
if segment.start < trim_before_secs {
segment.start = trim_before_secs;
}
}
}
fn transcribe_samples_sync(
engine: Arc<kon_transcription::LocalEngine>,
engine_name: &str,
samples: Vec<f32>,
options: TranscriptionOptions,
) -> Result<kon_transcription::TimedTranscript, String> {
let Some(strategy) = pick_chunking_strategy(engine_name, samples.len()) else {
let audio = AudioSamples::mono_16khz(samples);
return engine
.transcribe_sync(&audio, &options)
.map_err(|e| e.to_string());
};
let total_duration_secs = samples.len() as f64 / WHISPER_SAMPLE_RATE as f64;
let stride = strategy
.chunk_samples
.saturating_sub(strategy.overlap_samples)
.max(1);
let chunk_count = ((samples.len().saturating_sub(1)) / stride) + 1;
eprintln!(
"[transcription] chunking {total_duration_secs:.2}s of {engine_name} audio into {chunk_count} chunk(s)"
);
let mut all_segments = Vec::new();
let mut total_inference_ms = 0u64;
let mut chunk_start = 0usize;
while chunk_start < samples.len() {
let chunk_end = (chunk_start + strategy.chunk_samples).min(samples.len());
let chunk_audio = AudioSamples::mono_16khz(samples[chunk_start..chunk_end].to_vec());
let timed = engine
.transcribe_sync(&chunk_audio, &options)
.map_err(|e| e.to_string())?;
total_inference_ms = total_inference_ms.saturating_add(timed.inference_ms);
let mut chunk_segments = timed.transcript.segments().to_vec();
if chunk_start > 0 {
trim_overlap_segments(
&mut chunk_segments,
strategy.overlap_samples as f64 / WHISPER_SAMPLE_RATE as f64,
);
}
let chunk_offset_secs = chunk_start as f64 / WHISPER_SAMPLE_RATE as f64;
for segment in &mut chunk_segments {
segment.start += chunk_offset_secs;
segment.end += chunk_offset_secs;
}
all_segments.extend(chunk_segments);
if chunk_end >= samples.len() {
break;
}
chunk_start = chunk_end.saturating_sub(strategy.overlap_samples);
}
Ok(kon_transcription::TimedTranscript {
transcript: Transcript::new(
all_segments,
options.language.clone().unwrap_or_else(|| "en".to_string()),
total_duration_secs,
),
inference_ms: total_inference_ms,
})
}
/// Transcribe raw PCM f32 samples (Whisper). Emits "transcription-result" event.
#[tauri::command]
pub async fn transcribe_pcm(
@@ -38,8 +149,8 @@ pub async fn transcribe_pcm(
format_mode: String,
profile_id: Option<String>,
) -> Result<(), String> {
let resolved_profile_id = profile_id
.unwrap_or_else(|| kon_storage::DEFAULT_PROFILE_ID.to_string());
let resolved_profile_id =
profile_id.unwrap_or_else(|| kon_storage::DEFAULT_PROFILE_ID.to_string());
let profile = kon_storage::database::get_profile(&state.db, &resolved_profile_id)
.await
@@ -61,7 +172,6 @@ pub async fn transcribe_pcm(
};
let engine = state.whisper_engine.clone();
let audio = kon_core::AudioSamples::mono_16khz(samples);
let options = TranscriptionOptions {
language: Some(language),
initial_prompt: if effective_prompt.is_empty() {
@@ -72,7 +182,10 @@ pub async fn transcribe_pcm(
};
let timed = tokio::task::spawn_blocking(move || {
engine.transcribe_sync(&audio, &options).map_err(|e| e.to_string())
let audio = AudioSamples::mono_16khz(samples);
engine
.transcribe_sync(&audio, &options)
.map_err(|e| e.to_string())
})
.await
.map_err(|e| e.to_string())??;
@@ -122,8 +235,8 @@ pub async fn transcribe_file(
format_mode: String,
profile_id: Option<String>,
) -> Result<serde_json::Value, String> {
let resolved_profile_id = profile_id
.unwrap_or_else(|| kon_storage::DEFAULT_PROFILE_ID.to_string());
let resolved_profile_id =
profile_id.unwrap_or_else(|| kon_storage::DEFAULT_PROFILE_ID.to_string());
let profile = kon_storage::database::get_profile(&state.db, &resolved_profile_id)
.await
@@ -145,8 +258,8 @@ pub async fn transcribe_file(
};
let engine_name = engine.unwrap_or_else(|| "whisper".to_string());
let model_id = model_id
.unwrap_or_else(|| default_model_id_for_engine(&engine_name).to_string());
let model_id =
model_id.unwrap_or_else(|| default_model_id_for_engine(&engine_name).to_string());
ensure_model_loaded(&state, &engine_name, &model_id).await?;
let engine = pick_engine(&state, &engine_name)?;
@@ -158,15 +271,17 @@ pub async fn transcribe_file(
Some(effective_prompt)
},
};
let engine_name_for_worker = engine_name.clone();
let timed = tokio::task::spawn_blocking(move || {
let audio = kon_audio::decode_audio_file(Path::new(&path))
.map_err(|e| e.to_string())?;
let resampled =
kon_audio::resample_to_16khz(&audio).map_err(|e| e.to_string())?;
engine
.transcribe_sync(&resampled, &options)
.map_err(|e| e.to_string())
let audio = kon_audio::decode_audio_file(Path::new(&path)).map_err(|e| e.to_string())?;
let resampled = kon_audio::resample_to_16khz(&audio).map_err(|e| e.to_string())?;
transcribe_samples_sync(
engine,
&engine_name_for_worker,
resampled.into_samples(),
options,
)
})
.await
.map_err(|e| e.to_string())??;
@@ -208,8 +323,8 @@ pub async fn transcribe_pcm_parakeet(
format_mode: String,
profile_id: Option<String>,
) -> Result<(), String> {
let resolved_profile_id = profile_id
.unwrap_or_else(|| kon_storage::DEFAULT_PROFILE_ID.to_string());
let resolved_profile_id =
profile_id.unwrap_or_else(|| kon_storage::DEFAULT_PROFILE_ID.to_string());
// Validate the profile exists so parakeet and whisper behave identically
// when a bogus id slips through from the frontend.
@@ -227,11 +342,10 @@ pub async fn transcribe_pcm_parakeet(
.collect();
let engine = state.parakeet_engine.clone();
let audio = kon_core::AudioSamples::mono_16khz(samples);
let options = TranscriptionOptions::default();
let timed = tokio::task::spawn_blocking(move || {
engine.transcribe_sync(&audio, &options).map_err(|e| e.to_string())
transcribe_samples_sync(engine, "parakeet", samples, options)
})
.await
.map_err(|e| e.to_string())??;

20
src-tauri/src/commands/transcripts.rs
View File

@@ -17,8 +17,8 @@ use kon_storage::{
count_transcripts, delete_transcript as db_delete_transcript,
get_transcript as db_get_transcript, insert_transcript as db_insert_transcript,
list_transcripts_paged, search_transcripts as db_search_transcripts,
update_transcript as db_update_transcript,
update_transcript_meta as db_update_transcript_meta, InsertTranscriptParams, TranscriptRow,
update_transcript as db_update_transcript, update_transcript_meta as db_update_transcript_meta,
InsertTranscriptParams, TranscriptRow,
};
use crate::AppState;
@@ -36,6 +36,7 @@ pub struct TranscriptDto {
pub id: String,
pub text: String,
pub source: String,
pub profile_id: String,
pub title: Option<String>,
pub audio_path: Option<String>,
pub duration: f64,
@@ -55,6 +56,7 @@ impl From<TranscriptRow> for TranscriptDto {
id: r.id,
text: r.text,
source: r.source,
profile_id: r.profile_id,
title: r.title,
audio_path: r.audio_path,
duration: r.duration,
@@ -76,6 +78,7 @@ pub struct CreateTranscriptRequest {
pub id: String,
pub text: String,
pub source: String,
pub profile_id: Option<String>,
pub title: Option<String>,
pub audio_path: Option<String>,
pub duration: f64,
@@ -102,6 +105,10 @@ pub async fn add_transcript(
id: &transcript.id,
text: &transcript.text,
source: &transcript.source,
profile_id: transcript
.profile_id
.as_deref()
.unwrap_or(kon_storage::DEFAULT_PROFILE_ID),
title: transcript.title.as_deref(),
audio_path: transcript.audio_path.as_deref(),
duration: transcript.duration,
@@ -137,10 +144,10 @@ pub async fn list_transcripts(
/// Total count of transcripts (for "showing X of N" UI).
#[tauri::command]
pub async fn count_transcripts_command(
state: tauri::State<'_, AppState>,
) -> Result<i64, String> {
count_transcripts(&state.db).await.map_err(|e| e.to_string())
pub async fn count_transcripts_command(state: tauri::State<'_, AppState>) -> Result<i64, String> {
count_transcripts(&state.db)
.await
.map_err(|e| e.to_string())
}
#[tauri::command]
@@ -235,4 +242,3 @@ pub async fn update_transcript_meta_cmd(
.map_err(|e| e.to_string())?;
Ok(TranscriptDto::from(row))
}

44
src-tauri/src/commands/windows.rs
View File

@@ -4,9 +4,7 @@ use crate::PreferencesScript;
/// Open a floating always-on-top task window.
#[tauri::command]
pub async fn open_task_window(
app: tauri::AppHandle,
) -> Result<(), String> {
pub async fn open_task_window(app: tauri::AppHandle) -> Result<(), String> {
if let Some(window) = app.get_webview_window("tasks-float") {
window.show().map_err(|e| e.to_string())?;
window.set_focus().map_err(|e| e.to_string())?;
@@ -22,17 +20,14 @@ pub async fn open_task_window(
// custom frameless chrome drawn by the Titlebar component.
let use_native_decorations = cfg!(target_os = "linux");
let mut builder = WebviewWindowBuilder::new(
&app,
"tasks-float",
WebviewUrl::App("/float".into()),
)
.title("Kon Tasks")
.inner_size(480.0, 520.0)
.min_inner_size(360.0, 480.0)
.always_on_top(true)
.decorations(use_native_decorations)
.resizable(true);
let mut builder =
WebviewWindowBuilder::new(&app, "tasks-float", WebviewUrl::App("/float".into()))
.title("Kon Tasks")
.inner_size(480.0, 520.0)
.min_inner_size(360.0, 480.0)
.always_on_top(true)
.decorations(use_native_decorations)
.resizable(true);
// Inject preferences before Svelte mounts
if let Some(script) = app.try_state::<PreferencesScript>() {
@@ -48,9 +43,7 @@ pub async fn open_task_window(
/// Open the transcript viewer window.
#[tauri::command]
pub async fn open_viewer_window(
app: tauri::AppHandle,
) -> Result<(), String> {
pub async fn open_viewer_window(app: tauri::AppHandle) -> Result<(), String> {
if let Some(window) = app.get_webview_window("transcript-viewer") {
window.show().map_err(|e| e.to_string())?;
window.set_focus().map_err(|e| e.to_string())?;
@@ -60,16 +53,13 @@ pub async fn open_viewer_window(
// See note in open_task_window for the Linux-vs-other platform split.
let use_native_decorations = cfg!(target_os = "linux");
let mut builder = WebviewWindowBuilder::new(
&app,
"transcript-viewer",
WebviewUrl::App("/viewer".into()),
)
.title("Kon - Transcription Editor")
.inner_size(600.0, 700.0)
.min_inner_size(560.0, 520.0)
.decorations(use_native_decorations)
.resizable(true);
let mut builder =
WebviewWindowBuilder::new(&app, "transcript-viewer", WebviewUrl::App("/viewer".into()))
.title("Kon - Transcription Editor")
.inner_size(600.0, 700.0)
.min_inner_size(560.0, 520.0)
.decorations(use_native_decorations)
.resizable(true);
// Inject preferences before Svelte mounts
if let Some(script) = app.try_state::<PreferencesScript>() {

76
src-tauri/src/lib.rs
View File

@@ -100,7 +100,9 @@ fn ensure_x11_on_wayland() {
// SAFETY: setting env vars before any threads spawn (we are
// pre-Tauri-Builder here). This block is the only place these
// are written.
unsafe { std::env::set_var(key, value); }
unsafe {
std::env::set_var(key, value);
}
eprintln!("[startup] Wayland workaround: {key}={value}");
}
};
@@ -130,10 +132,8 @@ pub fn run() {
// Initialise database (blocking in setup — runs once at startup)
let db_path = database_path();
let t0 = Instant::now();
let db = tauri::async_runtime::block_on(async {
init_db(&db_path).await
})
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
let db = tauri::async_runtime::block_on(async { init_db(&db_path).await })
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
eprintln!("[startup] DB init: {:?}", t0.elapsed());
// Load saved preferences for webview injection
@@ -158,37 +158,40 @@ pub fn run() {
// signal and grant audio capture requests automatically.
#[cfg(target_os = "linux")]
{
main_window.with_webview(|webview| {
use webkit2gtk::{
PermissionRequest, PermissionRequestExt,
SettingsExt, WebViewExt,
};
main_window
.with_webview(|webview| {
use webkit2gtk::{
PermissionRequest, PermissionRequestExt, SettingsExt, WebViewExt,
};
let wv: webkit2gtk::WebView = webview.inner().clone();
let wv: webkit2gtk::WebView = webview.inner().clone();
// Enable media stream in WebKit settings
if let Some(settings) = WebViewExt::settings(&wv) {
settings.set_enable_media_stream(true);
settings.set_enable_media_capabilities(true);
}
// Enable media stream in WebKit settings
if let Some(settings) = WebViewExt::settings(&wv) {
settings.set_enable_media_stream(true);
settings.set_enable_media_capabilities(true);
}
// Auto-grant all permission requests (audio/video capture)
WebViewExt::connect_permission_request(&wv, |_wv, request: &PermissionRequest| {
request.allow();
true
// Auto-grant all permission requests (audio/video capture)
WebViewExt::connect_permission_request(
&wv,
|_wv, request: &PermissionRequest| {
request.allow();
true
},
);
})
.unwrap_or_else(|e| {
// Non-fatal: WebKitGTK may already have media
// capture wired by some compositors, or the
// signal binding may fail on unusual builds.
// Falling back means getUserMedia() prompts (or
// silently denies) instead of auto-granting,
// which is degraded but recoverable.
eprintln!(
"[startup] failed to configure webview media permissions: {e}",
);
});
})
.unwrap_or_else(|e| {
// Non-fatal: WebKitGTK may already have media
// capture wired by some compositors, or the
// signal binding may fail on unusual builds.
// Falling back means getUserMedia() prompts (or
// silently denies) instead of auto-granting,
// which is degraded but recoverable.
eprintln!(
"[startup] failed to configure webview media permissions: {e}",
);
});
}
// Close-to-tray: hide window instead of exiting
@@ -209,12 +212,8 @@ pub fn run() {
app.manage(commands::live::LiveTranscriptionState::default());
app.manage(AppState {
whisper_engine: Arc::new(LocalEngine::new(
EngineName::new("whisper"),
)),
parakeet_engine: Arc::new(LocalEngine::new(
EngineName::new("parakeet"),
)),
whisper_engine: Arc::new(LocalEngine::new(EngineName::new("whisper"))),
parakeet_engine: Arc::new(LocalEngine::new(EngineName::new("parakeet"))),
db,
llm_engine: Arc::new(LlmEngine::new()),
});
@@ -273,6 +272,7 @@ pub fn run() {
commands::profiles::delete_profile_cmd,
commands::profiles::list_profile_terms_cmd,
commands::profiles::add_profile_term_cmd,
commands::profiles::learn_profile_terms_from_edit_cmd,
commands::profiles::delete_profile_term_cmd,
// Transcripts (canonical SQLite-backed history) — Day 4
commands::transcripts::add_transcript,

4
src-tauri/src/tray.rs
View File

@@ -45,9 +45,7 @@ pub fn setup(app: &tauri::App) -> Result<(), Box<dyn std::error::Error>> {
..
} = event
{
if let Some(window) =
tray.app_handle().get_webview_window("main")
{
if let Some(window) = tray.app_handle().get_webview_window("main") {
let _ = window.show();
let _ = window.set_focus();
}