Lumotia/docs/architecture-map/03-audio-transcription/transcription-whisper.md

name, type, slice, last_verified

name	type	slice	last_verified
Whisper backend (whisper-rs)	architecture-map-page	03-audio-transcription	2026/05/09

Whisper backend (whisper-rs)

Where you are: Architecture map → Audio + Transcription → Whisper backend

Plain English summary. WhisperRsBackend owns a WhisperContext from whisper-rs 0.16 and runs Whisper directly (not via transcribe-rs). It is the only backend that pipes Lumotia's initial_prompt into the model. Each call to transcribe_sync builds a fresh WhisperState, applies language and prompt parameters, picks a thread count via the power-aware helper, and converts segment timestamps from centiseconds to seconds before returning Vec<Segment>. Vulkan GPU offload is additive behind a Cargo feature.

At a glance

• Crate: lumotia-transcription
• Path: crates/transcription/src/whisper_rs_backend.rs
• LOC: 128
• Cargo feature gate: whisper (default on). Module is excluded entirely when the feature is off.
• External deps: whisper-rs 0.16 (with optional vulkan sub-feature via whisper-vulkan), thiserror 2, tracing 0.1.
• Internal callers (best effort, slice 2 reconciles): local_engine::load_whisper (crates/transcription/src/local_engine.rs:208) constructs it; production load path runs through src-tauri/src/commands/models.rs.

Public surface:

• pub struct WhisperRsBackend — crates/transcription/src/whisper_rs_backend.rs:30. Field: ctx: WhisperContext.
• pub fn WhisperRsBackend::load(model_path: &Path) -> std::result::Result<Self, WhisperBackendError> — crates/transcription/src/whisper_rs_backend.rs:35.
• pub enum WhisperBackendError { Load(String), State(String), Transcribe(String) } — crates/transcription/src/whisper_rs_backend.rs:21. thiserror-derived; convertible to MagnotiaError::TranscriptionFailed via to_string().
• impl Transcriber for WhisperRsBackend — crates/transcription/src/whisper_rs_backend.rs:42.

What's in here

load

whisper_rs_backend.rs:35. WhisperContext::new_with_params(model_path, WhisperContextParameters::default()). Errors map to WhisperBackendError::Load. The context is the heavy object (mmap'd GGML tensors, GPU buffers if Vulkan is active). It is held for the life of the backend.

capabilities

whisper_rs_backend.rs:43. Returns sample_rate = WHISPER_SAMPLE_RATE (slice 5 constant), channels = 1, supports_initial_prompt = true. The truth flag is what tells the Settings UI to surface the prompt field.

transcribe_sync

whisper_rs_backend.rs:56. The actual inference path:

1. tracing::info! boundary log capturing language and has_initial_prompt — used by the diagnostic bundle to confirm the prompt actually reached the backend.
2. self.ctx.create_state() — fresh WhisperState per call. The header notes "state can be reused, but fresh-per-call is simpler and matches the transcribe-rs call style we are replacing".
3. FullParams::new(SamplingStrategy::Greedy { best_of: 1 }).
4. Optional params.set_language(Some(lang)) if options.language is set and non-empty.
5. Optional params.set_initial_prompt(prompt) if options.initial_prompt is set and non-empty.
6. params.set_n_threads(inference_thread_count(Workload::Whisper, gpu_offloaded) as i32) where gpu_offloaded = cfg!(feature = "whisper-vulkan") && vulkan_loader_available(). Power-aware (lumotia_core::tuning::inference_thread_count reads battery vs AC; helper returns a smaller count on battery to preserve runtime).
7. set_print_special(false), set_print_progress(false), set_print_realtime(false) — keep stdout silent.
8. state.full(params, samples) — runs inference.
9. Loops state.full_n_segments() calling state.get_segment(i), pulling seg.to_str(), and converting timestamps: start = seg.start_timestamp() as f64 * 0.01, end = seg.end_timestamp() as f64 * 0.01 (whisper-rs reports centiseconds).
10. Returns Vec<Segment>.

Errors all flow through MagnotiaError::TranscriptionFailed(WhisperBackendError::*.to_string()).

Data flow

samples (f32, 16 kHz mono)
options: TranscriptionOptions { language, initial_prompt }
  └─ tracing::info(boundary)
  └─ WhisperState (fresh per call)
  └─ FullParams (greedy, best_of=1)
       ├─ set_language(options.language)
       ├─ set_initial_prompt(options.initial_prompt)
       ├─ set_n_threads(inference_thread_count(Whisper, gpu_offloaded))
       └─ silence stdout flags
  └─ state.full(params, samples)
  └─ for i in 0..state.full_n_segments():
       Segment { start = t0 * 0.01, end = t1 * 0.01, text = seg.to_str() }
  → Vec<Segment>

Watch-outs

• whisper-rs-sys is the C++ link target. Brief item #6 documents the historical Whispering v7.11.0 failure linking whisper-rs-sys + tokenizers together on Windows MSVC. The build.rs guard panics if tokenizers ever lands in the workspace lockfile on a Windows target. See build-tokenizers-guard.md.
• Vulkan is detected at runtime, not just compile time. cfg!(feature = "whisper-vulkan") is the static check; vulkan_loader_available() (slice 5 lumotia_core::hardware) confirms libvulkan.so actually resolves. If the feature is on but the loader is missing, the helper falls back to CPU thread count.
• Fresh state per call has a cost. state.full warm-up is lower than WhisperContext load but non-zero. Brief comment hints reuse is possible. Worth measuring once the live-streaming path is dogfooded.
• No diarisation, no temperature fallback, no token suppression flags. Greedy with best_of: 1 is the simplest correct path. Tuning later is a single function.
• Timestamp units. whisper-rs returns centiseconds (10 ms). Multiplying by 0.01 gives seconds. A prior version that used milliseconds or token offsets would slice timing wrong; tests should pin this.
• initial_prompt is the differentiator from transcribe-rs. The whole point of writing this backend (vs going through SpeechModelAdapter) is that the adapter cannot pipe the prompt. Anyone tempted to "simplify" by routing Whisper through the adapter would silently drop the user-vocabulary feature.
• Tracing field name typo immune. language = ?options.language uses Debug formatting. If Language ever stops implementing Debug, this breaks at compile time, not at runtime.

See also

• Transcription engines overview — the Transcriber trait this implements.
• Cargo features — whisper and whisper-vulkan flag matrix.
• Build tokenizers guard — the Windows MSVC-CRT defence linked to this backend.
• Tests and fixtures — whisper_rs_smoke.rs exercises the load + transcribe + initial-prompt path; jfk_bench.rs measures cold/warm RTF; thread_sweep.rs validates thread-count scaling.
• lumotia_core::hardware::vulkan_loader_available, lumotia_core::tuning::{inference_thread_count, Workload} (slice 5).