Jake
26c7307607
Phase 9 of the rebrand cascade. Sweep covers everything the Phase 8
frontend pass deliberately skipped: docs/, root markdown, scripts,
Cargo.toml descriptions, code comments that survived earlier
word-boundary sed, plus a handful of identifiers caught on the final
verify pass.
transcription-app changes:
- README.md, HANDOVER.md, KNOWN-ISSUES.md, run.sh — magnotia/Magnotia
-> lumotia/Lumotia.
- docs/ — sweep across all subdirs except docs/handovers/ (preserved
as immutable audit trail). Includes architecture-map references
to magnotia_core::*, magnotia_storage::*, etc. now pointing at
lumotia_*; dev-setup.md tracing output examples (lumotia_startup
target); brief/ + superpowers/ + issues/ + whisper-ecosystem/ +
audit/.
- Cargo.toml descriptions on 9 crates (core, audio, cloud-providers,
hotkey, llm, mcp, plus referenced others).
- crates/core/src/{error,hardware,recommendation,paths}.rs +
crates/audio/src/wav.rs + crates/llm/src/model_manager.rs +
crates/cloud-providers/src/keystore.rs + crates/mcp/src/lib.rs —
doc comments and a model-manager user-agent string.
- Caught on final pass: BroadcastChannel("magnotia_task_sync") -> ...
("lumotia_task_sync"); magnotia_locale i18n localStorage key
renamed + migration shim added; CSS keyframe names
magnotiaPulse / magnotiaBar / magnotiaFade renamed in the design-
system kit; magnotia_viewer_item / magnotia_viewer_mode handoff
keys renamed in HistoryPage + viewer/+page.svelte; src/assets/
wordmark.svg text.
- src-tauri/src/lib.rs comment cleanup ("magnotia era" was sed'd
to "lumotia era" earlier — restored).
Preserved (intentional):
- crates/core/src/paths.rs — keeps "magnotia" / "Magnotia" / ".magnotia"
legacy detection strings in legacy_and_target_paths() so the
migration shim can still find user data from the magnotia era.
- src/lib/stores/{page,focusTimer}.svelte.ts + src/lib/i18n/index.ts
— migration call sites reference the legacy magnotia keys
deliberately.
- docs/handovers/ — historical audit trail.
cargo build --workspace passes. npm run check: 0 errors / 0 warnings
(3958 files). cargo test --workspace: 339 pass / 0 fail.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
92 lines
6.9 KiB
Markdown
92 lines
6.9 KiB
Markdown
---
|
|
name: Whisper backend (whisper-rs)
|
|
type: architecture-map-page
|
|
slice: 03-audio-transcription
|
|
last_verified: 2026/05/09
|
|
---
|
|
|
|
# Whisper backend (whisper-rs)
|
|
|
|
> **Where you are:** [Architecture map](../README.md) → [Audio + Transcription](README.md) → Whisper backend
|
|
|
|
**Plain English summary.** `WhisperRsBackend` owns a `WhisperContext` from `whisper-rs` 0.16 and runs Whisper directly (not via `transcribe-rs`). It is the only backend that pipes Lumotia's `initial_prompt` into the model. Each call to `transcribe_sync` builds a fresh `WhisperState`, applies language and prompt parameters, picks a thread count via the power-aware helper, and converts segment timestamps from centiseconds to seconds before returning `Vec<Segment>`. Vulkan GPU offload is additive behind a Cargo feature.
|
|
|
|
## At a glance
|
|
|
|
- Crate: `lumotia-transcription`
|
|
- Path: `crates/transcription/src/whisper_rs_backend.rs`
|
|
- LOC: 128
|
|
- Cargo feature gate: `whisper` (default on). Module is excluded entirely when the feature is off.
|
|
- External deps: `whisper-rs 0.16` (with optional `vulkan` sub-feature via `whisper-vulkan`), `thiserror 2`, `tracing 0.1`.
|
|
- Internal callers (best effort, slice 2 reconciles): `local_engine::load_whisper` (`crates/transcription/src/local_engine.rs:208`) constructs it; production load path runs through `src-tauri/src/commands/models.rs`.
|
|
|
|
Public surface:
|
|
|
|
- `pub struct WhisperRsBackend` — `crates/transcription/src/whisper_rs_backend.rs:30`. Field: `ctx: WhisperContext`.
|
|
- `pub fn WhisperRsBackend::load(model_path: &Path) -> std::result::Result<Self, WhisperBackendError>` — `crates/transcription/src/whisper_rs_backend.rs:35`.
|
|
- `pub enum WhisperBackendError { Load(String), State(String), Transcribe(String) }` — `crates/transcription/src/whisper_rs_backend.rs:21`. `thiserror`-derived; convertible to `MagnotiaError::TranscriptionFailed` via `to_string()`.
|
|
- `impl Transcriber for WhisperRsBackend` — `crates/transcription/src/whisper_rs_backend.rs:42`.
|
|
|
|
## What's in here
|
|
|
|
### `load`
|
|
|
|
`whisper_rs_backend.rs:35`. `WhisperContext::new_with_params(model_path, WhisperContextParameters::default())`. Errors map to `WhisperBackendError::Load`. The context is the heavy object (mmap'd GGML tensors, GPU buffers if Vulkan is active). It is held for the life of the backend.
|
|
|
|
### `capabilities`
|
|
|
|
`whisper_rs_backend.rs:43`. Returns `sample_rate = WHISPER_SAMPLE_RATE` (slice 5 constant), `channels = 1`, `supports_initial_prompt = true`. The truth flag is what tells the Settings UI to surface the prompt field.
|
|
|
|
### `transcribe_sync`
|
|
|
|
`whisper_rs_backend.rs:56`. The actual inference path:
|
|
|
|
1. `tracing::info!` boundary log capturing `language` and `has_initial_prompt` — used by the diagnostic bundle to confirm the prompt actually reached the backend.
|
|
2. `self.ctx.create_state()` — fresh `WhisperState` per call. The header notes "state can be reused, but fresh-per-call is simpler and matches the transcribe-rs call style we are replacing".
|
|
3. `FullParams::new(SamplingStrategy::Greedy { best_of: 1 })`.
|
|
4. Optional `params.set_language(Some(lang))` if `options.language` is set and non-empty.
|
|
5. Optional `params.set_initial_prompt(prompt)` if `options.initial_prompt` is set and non-empty.
|
|
6. `params.set_n_threads(inference_thread_count(Workload::Whisper, gpu_offloaded) as i32)` where `gpu_offloaded = cfg!(feature = "whisper-vulkan") && vulkan_loader_available()`. Power-aware (`lumotia_core::tuning::inference_thread_count` reads battery vs AC; helper returns a smaller count on battery to preserve runtime).
|
|
7. `set_print_special(false)`, `set_print_progress(false)`, `set_print_realtime(false)` — keep stdout silent.
|
|
8. `state.full(params, samples)` — runs inference.
|
|
9. Loops `state.full_n_segments()` calling `state.get_segment(i)`, pulling `seg.to_str()`, and converting timestamps: `start = seg.start_timestamp() as f64 * 0.01`, `end = seg.end_timestamp() as f64 * 0.01` (whisper-rs reports centiseconds).
|
|
10. Returns `Vec<Segment>`.
|
|
|
|
Errors all flow through `MagnotiaError::TranscriptionFailed(WhisperBackendError::*.to_string())`.
|
|
|
|
## Data flow
|
|
|
|
```
|
|
samples (f32, 16 kHz mono)
|
|
options: TranscriptionOptions { language, initial_prompt }
|
|
└─ tracing::info(boundary)
|
|
└─ WhisperState (fresh per call)
|
|
└─ FullParams (greedy, best_of=1)
|
|
├─ set_language(options.language)
|
|
├─ set_initial_prompt(options.initial_prompt)
|
|
├─ set_n_threads(inference_thread_count(Whisper, gpu_offloaded))
|
|
└─ silence stdout flags
|
|
└─ state.full(params, samples)
|
|
└─ for i in 0..state.full_n_segments():
|
|
Segment { start = t0 * 0.01, end = t1 * 0.01, text = seg.to_str() }
|
|
→ Vec<Segment>
|
|
```
|
|
|
|
## Watch-outs
|
|
|
|
- **`whisper-rs-sys` is the C++ link target.** Brief item #6 documents the historical Whispering v7.11.0 failure linking `whisper-rs-sys` + `tokenizers` together on Windows MSVC. The `build.rs` guard panics if `tokenizers` ever lands in the workspace lockfile on a Windows target. See [build-tokenizers-guard.md](build-tokenizers-guard.md).
|
|
- **Vulkan is detected at runtime, not just compile time.** `cfg!(feature = "whisper-vulkan")` is the static check; `vulkan_loader_available()` (slice 5 `lumotia_core::hardware`) confirms `libvulkan.so` actually resolves. If the feature is on but the loader is missing, the helper falls back to CPU thread count.
|
|
- **Fresh state per call has a cost.** `state.full` warm-up is lower than `WhisperContext` load but non-zero. Brief comment hints reuse is possible. Worth measuring once the live-streaming path is dogfooded.
|
|
- **No diarisation, no temperature fallback, no token suppression flags.** Greedy with `best_of: 1` is the simplest correct path. Tuning later is a single function.
|
|
- **Timestamp units.** whisper-rs returns centiseconds (10 ms). Multiplying by `0.01` gives seconds. A prior version that used milliseconds or token offsets would slice timing wrong; tests should pin this.
|
|
- **`initial_prompt` is the differentiator from `transcribe-rs`.** The whole point of writing this backend (vs going through `SpeechModelAdapter`) is that the adapter cannot pipe the prompt. Anyone tempted to "simplify" by routing Whisper through the adapter would silently drop the user-vocabulary feature.
|
|
- **Tracing field name typo immune.** `language = ?options.language` uses Debug formatting. If `Language` ever stops implementing `Debug`, this breaks at compile time, not at runtime.
|
|
|
|
## See also
|
|
|
|
- [Transcription engines overview](transcription-engines-overview.md) — the `Transcriber` trait this implements.
|
|
- [Cargo features](cargo-features.md) — `whisper` and `whisper-vulkan` flag matrix.
|
|
- [Build tokenizers guard](build-tokenizers-guard.md) — the Windows MSVC-CRT defence linked to this backend.
|
|
- [Tests and fixtures](tests-and-fixtures.md) — `whisper_rs_smoke.rs` exercises the load + transcribe + initial-prompt path; `jfk_bench.rs` measures cold/warm RTF; `thread_sweep.rs` validates thread-count scaling.
|
|
- `lumotia_core::hardware::vulkan_loader_available`, `lumotia_core::tuning::{inference_thread_count, Workload}` (slice 5).
|