feat(transcription A.1 #23): silent warm-up inference after Whisper model load
Some checks failed
check / cargo check (macos-latest) (push) Has been cancelled
check / cargo check (ubuntu-22.04) (push) Has been cancelled
check / cargo check (windows-latest) (push) Has been cancelled
check / svelte build + lint (push) Has been cancelled

prewarm_default_model loaded the model and returned. That moves the
model into RAM, but whisper.cpp still allocates its context window +
fills GPU shader caches on the first inference call — producing the
~4–5 s cold-start latency documented in ufal/whisper_streaming #96
and #135 that feels like "Kon dropped my first sentence."

Extend the pre-warm task: after engine.load, feed one second of
silence (16000 zero samples at 16 kHz) through transcribe_sync with
default options. Silence returns empty segments; the *work* is the
context allocation, which now happens at app boot rather than on the
user's first hotkey press.

Net: the user's first real dictation should complete within ~1.5× the
steady-state RTF they'll see on subsequent runs, satisfying the A.1
#23 acceptance criterion. No new public API; all inside the existing
spawn_blocking.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-04-21 16:25:09 +01:00
parent 42ba18a274
commit e10f435eb1

View File

@@ -4,9 +4,10 @@ use serde::Serialize;
use tauri::Emitter;
use crate::AppState;
use kon_core::constants::WHISPER_SAMPLE_RATE;
use kon_core::hardware::{self, CpuFeatures};
use kon_core::model_registry::{self, Engine, LanguageSupport, ModelEntry};
use kon_core::types::ModelId;
use kon_core::types::{AudioSamples, ModelId, TranscriptionOptions};
use kon_transcription::model_manager;
use kon_transcription::{load_parakeet, load_whisper, LocalEngine};
@@ -179,6 +180,20 @@ pub fn prewarm_default_model(whisper_engine: Arc<LocalEngine>) {
let result = tauri::async_runtime::spawn_blocking(move || {
load_model_from_disk(&model_id).map(|model| {
whisper_engine.load(model, model_id);
// Silent warm-up pass: feed one second of silence through
// the freshly-loaded engine. Pre-allocates the Whisper
// context window + warms GPU shader caches so the user's
// first real transcription completes in ≤1.5× steady-state
// latency instead of the ~45s cold-start documented in
// ufal/whisper_streaming #96 and #135. Silence returns
// empty segments — the *work* is the context allocation.
let silence =
AudioSamples::mono_16khz(vec![0.0_f32; WHISPER_SAMPLE_RATE as usize]);
let options = TranscriptionOptions::default();
match whisper_engine.transcribe_sync(&silence, &options) {
Ok(_) => eprintln!("[startup] Whisper warm-up inference complete"),
Err(e) => eprintln!("[startup] Whisper warm-up inference failed: {e}"),
}
})
})
.await;