---
name: Whisper backend (whisper-rs)
type: architecture-map-page
slice: 03-audio-transcription
last_verified: 2026/05/09
---

# Whisper backend (whisper-rs)

> **Where you are:** [Architecture map](../README.md) → [Audio + Transcription](README.md) → Whisper backend

**Plain English summary.** `WhisperRsBackend` owns a `WhisperContext` from `whisper-rs` 0.16 and runs Whisper directly (not via `transcribe-rs`). It is the only backend that pipes Lumotia's `initial_prompt` into the model. Each call to `transcribe_sync` builds a fresh `WhisperState`, applies language and prompt parameters, picks a thread count via the power-aware helper, and converts segment timestamps from centiseconds to seconds before returning `Vec<Segment>`. Vulkan GPU offload is additive behind a Cargo feature.

## At a glance

- Crate: `lumotia-transcription`
- Path: `crates/transcription/src/whisper_rs_backend.rs`
- LOC: 128
- Cargo feature gate: `whisper` (default on). Module is excluded entirely when the feature is off.
- External deps: `whisper-rs 0.16` (with optional `vulkan` sub-feature via `whisper-vulkan`), `thiserror 2`, `tracing 0.1`.
- Internal callers (best effort, slice 2 reconciles): `local_engine::load_whisper` (`crates/transcription/src/local_engine.rs:208`) constructs it; production load path runs through `src-tauri/src/commands/models.rs`.

Public surface:

- `pub struct WhisperRsBackend` — `crates/transcription/src/whisper_rs_backend.rs:30`. Field: `ctx: WhisperContext`.
- `pub fn WhisperRsBackend::load(model_path: &Path) -> std::result::Result<Self, WhisperBackendError>` — `crates/transcription/src/whisper_rs_backend.rs:35`.
- `pub enum WhisperBackendError { Load(String), State(String), Transcribe(String) }` — `crates/transcription/src/whisper_rs_backend.rs:21`. `thiserror`-derived; convertible to `MagnotiaError::TranscriptionFailed` via `to_string()`.
- `impl Transcriber for WhisperRsBackend` — `crates/transcription/src/whisper_rs_backend.rs:42`.

## What's in here

### `load`

`whisper_rs_backend.rs:35`. `WhisperContext::new_with_params(model_path, WhisperContextParameters::default())`. Errors map to `WhisperBackendError::Load`. The context is the heavy object (mmap'd GGML tensors, GPU buffers if Vulkan is active). It is held for the life of the backend.

### `capabilities`

`whisper_rs_backend.rs:43`. Returns `sample_rate = WHISPER_SAMPLE_RATE` (slice 5 constant), `channels = 1`, `supports_initial_prompt = true`. The truth flag is what tells the Settings UI to surface the prompt field.

### `transcribe_sync`

`whisper_rs_backend.rs:56`. The actual inference path:

1. `tracing::info!` boundary log capturing `language` and `has_initial_prompt` — used by the diagnostic bundle to confirm the prompt actually reached the backend.
2. `self.ctx.create_state()` — fresh `WhisperState` per call. The header notes "state can be reused, but fresh-per-call is simpler and matches the transcribe-rs call style we are replacing".
3. `FullParams::new(SamplingStrategy::Greedy { best_of: 1 })`.
4. Optional `params.set_language(Some(lang))` if `options.language` is set and non-empty.
5. Optional `params.set_initial_prompt(prompt)` if `options.initial_prompt` is set and non-empty.
6. `params.set_n_threads(inference_thread_count(Workload::Whisper, gpu_offloaded) as i32)` where `gpu_offloaded = cfg!(feature = "whisper-vulkan") && vulkan_loader_available()`. Power-aware (`lumotia_core::tuning::inference_thread_count` reads battery vs AC; helper returns a smaller count on battery to preserve runtime).
7. `set_print_special(false)`, `set_print_progress(false)`, `set_print_realtime(false)` — keep stdout silent.
8. `state.full(params, samples)` — runs inference.
9. Loops `state.full_n_segments()` calling `state.get_segment(i)`, pulling `seg.to_str()`, and converting timestamps: `start = seg.start_timestamp() as f64 * 0.01`, `end = seg.end_timestamp() as f64 * 0.01` (whisper-rs reports centiseconds).
10. Returns `Vec<Segment>`.

Errors all flow through `MagnotiaError::TranscriptionFailed(WhisperBackendError::*.to_string())`.

## Data flow

```
samples (f32, 16 kHz mono)
options: TranscriptionOptions { language, initial_prompt }
  └─ tracing::info(boundary)
  └─ WhisperState (fresh per call)
  └─ FullParams (greedy, best_of=1)
       ├─ set_language(options.language)
       ├─ set_initial_prompt(options.initial_prompt)
       ├─ set_n_threads(inference_thread_count(Whisper, gpu_offloaded))
       └─ silence stdout flags
  └─ state.full(params, samples)
  └─ for i in 0..state.full_n_segments():
       Segment { start = t0 * 0.01, end = t1 * 0.01, text = seg.to_str() }
  → Vec<Segment>
```

## Watch-outs

- **`whisper-rs-sys` is the C++ link target.** Brief item #6 documents the historical Whispering v7.11.0 failure linking `whisper-rs-sys` + `tokenizers` together on Windows MSVC. The `build.rs` guard panics if `tokenizers` ever lands in the workspace lockfile on a Windows target. See [build-tokenizers-guard.md](build-tokenizers-guard.md).
- **Vulkan is detected at runtime, not just compile time.** `cfg!(feature = "whisper-vulkan")` is the static check; `vulkan_loader_available()` (slice 5 `lumotia_core::hardware`) confirms `libvulkan.so` actually resolves. If the feature is on but the loader is missing, the helper falls back to CPU thread count.
- **Fresh state per call has a cost.** `state.full` warm-up is lower than `WhisperContext` load but non-zero. Brief comment hints reuse is possible. Worth measuring once the live-streaming path is dogfooded.
- **No diarisation, no temperature fallback, no token suppression flags.** Greedy with `best_of: 1` is the simplest correct path. Tuning later is a single function.
- **Timestamp units.** whisper-rs returns centiseconds (10 ms). Multiplying by `0.01` gives seconds. A prior version that used milliseconds or token offsets would slice timing wrong; tests should pin this.
- **`initial_prompt` is the differentiator from `transcribe-rs`.** The whole point of writing this backend (vs going through `SpeechModelAdapter`) is that the adapter cannot pipe the prompt. Anyone tempted to "simplify" by routing Whisper through the adapter would silently drop the user-vocabulary feature.
- **Tracing field name typo immune.** `language = ?options.language` uses Debug formatting. If `Language` ever stops implementing `Debug`, this breaks at compile time, not at runtime.

## See also

- [Transcription engines overview](transcription-engines-overview.md) — the `Transcriber` trait this implements.
- [Cargo features](cargo-features.md) — `whisper` and `whisper-vulkan` flag matrix.
- [Build tokenizers guard](build-tokenizers-guard.md) — the Windows MSVC-CRT defence linked to this backend.
- [Tests and fixtures](tests-and-fixtures.md) — `whisper_rs_smoke.rs` exercises the load + transcribe + initial-prompt path; `jfk_bench.rs` measures cold/warm RTF; `thread_sweep.rs` validates thread-count scaling.
- `lumotia_core::hardware::vulkan_loader_available`, `lumotia_core::tuning::{inference_thread_count, Workload}` (slice 5).