jake/Lumotia
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(phase4-systems): execution plan for Workstream A

Browse Source 
Sequences the 18 A-scope items from docs/whisper-ecosystem/brief.md
into four phases (pre-emptive patches, engine abstraction, streaming
correctness, LLM guard) with stop-for-review boundaries between each.

Lists the new command + event contracts (#1 activeComputeDevice,
#14 list_gpus/set_preferred_gpu, #24 tentative segment flag, #28
parallel-mode toggle) that Workstream B will consume.

Co-authored-by: jars <jakejars@users.noreply.github.com>
This commit is contained in:
Cursor Agent
2026-04-21 10:55:18 +00:00
committed by Jake
parent da74a84009
commit 35cfdfddf1
1 changed files with 270 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

270
docs/whisper-ecosystem/workstream-A.md Normal file
View File

@@ -0,0 +1,270 @@
# Workstream A — Systems hardening + streaming correctness
*Execution plan for the Codex half of the Whisper-ecosystem pass.*
*Branch: `phase4-systems-f7d0` (base: `main`). Companion: `phase4-ux-f7d0` (Workstream B).*
This is the backend / systems half. Every task below maps to an atomic
item in `docs/whisper-ecosystem/brief.md`. Items #3, #4 (UX side only),
#5, #10, #11, #14 (UI), #15, #16, #17, #20, #27, #30, #31 are owned by
Workstream B and are listed here only where the contract is shared.
---
## Scope recap
**In scope for A:** items #1, #2, #6, #7, #8, #9, #12, #13, #18, #19,
#21, #22, #23, #24, #25, #26, #28, #29.
**Explicit skip:** #18 (initial-prompt priming) is already shipped —
`src-tauri/src/commands/mod.rs::build_initial_prompt` plus
`crates/transcription/src/whisper_rs_backend.rs` already wire the
request prompt + profile prompt + profile vocabulary into
`FullParams::set_initial_prompt`. Workstream A verifies via the existing
lib tests and moves on; no new code.
**Untouchable (owned by B):** `src/lib/pages/*.svelte`, `src/routes/**`,
`crates/ai-formatting/src/llm_client.rs` (B may only edit
`CLEANUP_PROMPT` on that file).
**Untouchable (global):** the `ggml` dedup workaround in
`src-tauri/build.rs` is pinned as a known interim — do not touch.
---
## Execution order
The backlog groups itself into four natural phases. Each phase lands as
a sequence of small, one-concern commits, with `cargo test --workspace
--lib && cargo build -p kon && npm run check` green at every commit
boundary. At the end of each phase we pause for review before the next.
### Phase A.1 — Pre-emptive patches (no new UX surface)
| Seq | Item | Effect | Depends on |
|---|---|---|---|
| 1 | **#2**: pre-approve `http://127.0.0.1:*` in Tauri capabilities for LLM | Adds `http:default-scope-127.0.0.1` style capability + widens `connect-src` CSP to localhost | — |
| 2 | **#6**: guard `whisper-rs-sys` + `tokenizers` on Windows | Audit: Kon doesn't link `tokenizers` directly; we add a `#[cfg(all(target_os = "windows"))]` compile-time assert in `kon-transcription/build.rs` that fails the build if anything in the tree pulls in `tokenizers` on Windows | — |
| 3 | **#7**: CPU feature detection + non-AVX2 fallback surface | Extend `crates/core/src/hardware.rs` with `CpuFeatures` (avx2, avx512, fma, sse4_2) via `std::is_x86_feature_detected!` at runtime; surface as part of `RuntimeCapabilities`; when AVX2 is absent, emit a `warning` log + `runtime-warning` event so the frontend can show a banner | #1 |
| 4 | **#1**: detect and surface active compute device | Extend `RuntimeCapabilities` with an `ActiveComputeDevice` struct (`kind: "cuda" \| "vulkan" \| "metal" \| "cpu"`, `label: String`, `reason: Option<String>`). For Whisper, read the first `whisper_print_system_info` line via a shim in `whisper_rs_backend`; reason is filled on any CPU fallback (e.g., "Vulkan loader not found") | #7 |
| 5 | **#8**: checksum + resumable model downloads; retain audio on failure | Port the `crates/llm/src/model_manager.rs` pattern (SHA256 verify, `.part` file resume, `ResumeUnsupported` error) into `crates/transcription/src/model_manager.rs::download_file`. The existing code already does incremental SHA256 and range resume but lacks (a) validation of an **existing full file** before re-downloading, (b) ResumeUnsupported signalling, (c) test coverage parity. Retaining audio on failure is already covered in `commands/live.rs` via `save_audio` — add an explicit retry-friendly error classification so the frontend can render "Retry transcription" (a B surface, but the error enum ships here) | — |
| 6 | **#9**: disable macOS App Nap during capture + LLM | Add `commands/power.rs` with `cfg(target_os = "macos")` using `NSProcessInfo beginActivityWithOptions:reason:` to pin a power-assertion handle during live sessions and LLM generation | — |
| 7 | **#12**: bundle Vulkan loader + libssl on Windows, CPU fallback | Update `src-tauri/tauri.conf.json` bundle `resources` to list `vulkan-1.dll` and `libssl-3-x64.dll`/`libcrypto-3-x64.dll`; add a `cfg(target_os = "windows")` startup hook that probes `LoadLibraryW("vulkan-1.dll")` and downgrades `active_compute_device` to CPU with a clear reason if absent. Ship a pre-`cargo build` note in CI (README) — we cannot sign the installer from Linux CI | #1 |
**Commit boundary:** `cargo test --workspace --lib` (must include new
tests for checksum resume + CPU feature detect + active-device shim),
`cargo build -p kon`, `npm run check`. Then **stop for review.**
### Phase A.2 — Engine abstraction (#13, #19)
| Seq | Item | Effect | Depends on |
|---|---|---|---|
| 8 | **#13 (engine trait)**: `kon-transcription` already has `LocalEngine` and a `SpeechBackend` enum over `transcribe-rs` adapter + `WhisperRsBackend`. This is a 90% version of Handy's `transcribe-rs` pattern. We lift it one more inch: replace the enum with a public `Transcriber` trait (`load`, `transcribe_sync`, `capabilities`) + blanket impls for the existing two backends, and gate `WhisperRsBackend` behind the `whisper` feature so non-Whisper builds compile without pulling `whisper-rs-sys` | — |
| 9 | **#19**: progressive WAV write during live capture | Extend `commands/live.rs` to stream captured mono-16kHz samples directly into a `.wav` in the session folder via `kon_audio::WavWriter::append`. Replace the in-memory `kept_audio: Vec<f32>` when `save_audio` is on with a disk-backed writer; on crash, the partial file is a playable WAV. Commits the `commands/live.rs` frame-size bookkeeping untouched | #13 (uses `Transcriber::capabilities().sample_rate`) |
**Commit boundary:** tests (`transcriber_trait_is_object_safe`,
`wav_writer_survives_crash`), build, check. **Stop for review.**
### Phase A.3 — Streaming correctness (#21–#26)
This is the biggest change surface. We replace the custom RMS speech
gate in `commands/live.rs` with a Silero-VAD-gated chunker and layer
hallucination defences on top. Everything goes into a new
`crates/transcription/src/streaming/` module; `commands/live.rs`
becomes a thin driver.
| Seq | Item | Effect | Depends on |
|---|---|---|---|
| 10 | **#23**: warm-up silent WAV at app launch and after 60s idle | Add a tiny 1s silent-WAV loader (shipped in `src-tauri/assets/warmup.wav`) that runs through `LocalEngine::transcribe_sync` once after load_model; reschedules itself after any 60s idle gap in live sessions. Fixes the 4–5s cold-start latency cliff | #13 |
| 11 | **#21**: Silero-VAD-gated chunker | Add `silero-vad` (ort-backed, small ONNX model) as a dependency. New `streaming::VadChunker` takes a 16 kHz mono stream, emits `(chunk_start_sample, samples)` pairs when the VAD score crosses a threshold with hysteresis. Configurable threshold (default 0.5, hysteresis 0.35). Replaces the RMS-based `evaluate_speech_gate` for the chunk-boundary decision. Current `evaluate_speech_gate` stays as a _second-line_ silence skip | #19 (buffer ownership moved) |
| 12 | **#22**: hallucination blocklist + `avg_logprob`/`no_speech_prob` gate | Extend `WhisperRsBackend` to surface `no_speech_prob` + `avg_logprob` per segment (currently discarded). Add `streaming::HallucinationFilter` with a blocklist (static + user-editable via profile terms — blocked list, not vocab) and a confidence gate. Drop whole chunks that score below gate; drop segments whose text matches blocklist phrases | #21 |
| 13 | **#24**: LocalAgreement-n commit policy | Add `streaming::CommitPolicy::LocalAgreement { n: 2 }`. Two consecutive passes must produce matching prefix tokens before emission; tentative tail is sent with a `tentative: true` flag on `LiveResultMessage.segments` for the UI (B renders the dashed underline — contract below) | #22 |
| 14 | **#25**: aggressive buffer trim tied to commit points | Replace the current "drain OVERLAP_SAMPLES" with "drain everything up to the last commit point emitted by `CommitPolicy`". Guarantees the working buffer stays bounded regardless of wall-clock session length | #24 |
| 15 | **#26**: repetition detector + context window reset | Add `streaming::RepetitionDetector`: three consecutive identical token runs (per whisper-rs token, or per-word if running Parakeet) triggers `WhisperContext::create_state` + drop the offending chunk, logs a `runtime-warning` event | #22 |
**Commit boundary:** tests for the VAD chunker (with a fixture WAV that
contains silence → speech → silence), the LocalAgreement commit policy
(feed two fake passes, assert only the agreed prefix emits), and the
repetition detector. **Stop for review.**
### Phase A.4 — LLM guard (#28, #29)
| Seq | Item | Effect | Depends on |
|---|---|---|---|
| 16 | **#28**: sequential Whisper→LLM execution; single-GPU guard | Add `crates/transcription/src/concurrency.rs::GpuGuard` (already exists as file; extend it) — an async `Semaphore::new(1)` gate that wraps every `transcribe_sync` and every `LlmEngine::generate` on single-GPU systems. A `parallel_mode: bool` setting (default false; exposed via `get_runtime_capabilities` for B to wire into Settings) opens the gate to `n = 2` for users with ≥16 GB VRAM detected by `probe_gpu` | #1 |
| 17 | **#29**: plain-text pre-formatter before LLM prompt | `crates/ai-formatting/src/pipeline.rs` already joins segments; extend it to strip timestamps/IDs and normalise whitespace for the LLM call. New `crates/ai-formatting/src/to_plain_text.rs` with a pure function + tests | — |
**Commit boundary:** tests, build, check. **Stop for review.**
---
## Functions extended vs replaced
| Existing function / module | Extend or replace | Rationale |
|---|---|---|
| `commands::mod::build_initial_prompt` | **Keep as-is** | Already covers #18 |
| `commands::models::get_runtime_capabilities` | **Extend** | Add `active_compute_device`, `cpu_features`, `parallel_mode_available` fields |
| `commands::live::evaluate_speech_gate` | **Replace** with VAD-gated chunker, keep RMS version as a cheap fallback when VAD ONNX model missing |
| `commands::live::run_live_session` | **Extend** | Switch buffer ownership to `streaming::StreamingSession`, keep the `cpal` plumbing |
| `crates/transcription/src/model_manager::download_file` | **Extend** | Harden to match `kon-llm`'s resume + sha behaviour, add `ResumeUnsupported` arm |
| `crates/transcription/src/whisper_rs_backend::transcribe_sync` | **Extend** | Surface `no_speech_prob` + `avg_logprob` per segment |
| `crates/transcription/src/local_engine::SpeechBackend` | **Replace** with public `Transcriber` trait |
| `crates/transcription/src/concurrency` | **Extend** | Already a stub; add GPU guard wrapper |
| `crates/ai-formatting/src/pipeline` | **Extend** only the plain-text join path |
| `crates/core/src/hardware::probe_system` | **Extend** | Add `cpu_features` + flesh out `probe_gpu` (which currently returns `None`) with a best-effort Vulkan loader probe |
---
## New commands and event contracts (for Workstream B)
Workstream B needs stable surfaces to wire into. These are the only
Rust-side contracts A commits to in Phase A.1 and A.2 (Phase A.3/A.4
contracts are listed inline above):
### Item #1: active compute device
Extend `RuntimeCapabilities` (no new command):
```jsonc
{
"accelerators": ["cpu", "vulkan"],
"activeComputeDevice": {
"kind": "vulkan", // "cuda" | "vulkan" | "metal" | "cpu"
"label": "NVIDIA GeForce RTX 3060 (Vulkan)",
"reason": null // or "Vulkan loader not found, running on CPU"
},
"cpuFeatures": {
"avx2": true,
"avx512": false,
"fma": true,
"sse4_2": true
},
"parallelModeAvailable": false, // true only when >= 16 GB VRAM
"engines": [ /* unchanged */ ]
}
```
Event:
```
event: runtime-warning
payload: { kind: "avx2-missing" | "vulkan-loader-missing" | "cuda-fallback", message: string }
```
Emitted once at startup; B shows a dismissible Settings banner.
### Item #14: GPU enumeration (for B's explicit selector)
New command, for B to wire the Settings dropdown:
```
command: list_gpus()
returns: [
{
"id": "cuda:0",
"label": "NVIDIA GeForce RTX 3060",
"vram_mb": 12288,
"backend": "cuda", // "cuda" | "vulkan" | "metal" | "cpu"
"active": true
},
{
"id": "cpu",
"label": "CPU (Ryzen 7 7840U, 16 threads)",
"vram_mb": null,
"backend": "cpu",
"active": false
}
]
command: set_preferred_gpu(id: string)
returns: Ok(()) — persists to settings table, takes effect on next model load
```
Note that **actually swapping** the device requires rebuilding the
whisper-cpp backend with a different feature flag; for now this command
stores the preference, surfaces a "Restart to apply" toast, and
`prewarm_default_model` re-checks it on next launch.
### Item #19 / #23: warm-up and progressive WAV
No new frontend-visible command. The `save_audio` flag on
`start_live_transcription_session` already covers this; B should
continue to use it.
### Item #24: tentative vs committed segments (streaming)
`LiveResultMessage.segments[i]` gets a new field:
```
{
"start": 1.2,
"end": 1.8,
"text": "hello world",
"tentative": false // true when emitted under LocalAgreement tail
}
```
B is expected to render `tentative: true` with a dashed underline in
the preview overlay. Existing consumers that ignore unknown fields keep
working; this field is additive.
### Item #28: parallel-mode setting
Extend `SettingsState` on B's side to include
`parallelWhisperLlm: boolean` (default `false`). B wires this to a new
Rust command:
```
command: set_parallel_gpu_mode(enabled: bool) -> Ok(())
```
which flips the `GpuGuard`'s semaphore permit count. Gated in Settings
behind `runtimeCapabilities.parallelModeAvailable`.
---
## Test strategy
- Every phase ends green on `cargo test --workspace --lib && cargo
build -p kon && npm run check`.
- `phase4-systems-f7d0` never regresses the 116 existing lib tests. If
a test starts failing and the cause is an A change, it's fixed
in-phase, not deferred.
- New unit tests land with their phase: the VAD chunker has a WAV
fixture test, the LocalAgreement policy has a synthetic two-pass
test, the download resume already has a `tokio::net::TcpListener`
fixture test — we port the pattern.
- Manual smoke test after Phase A.3 on Jake's dev box: 10-minute live
dictation session, assert (a) no latency growth past chunk 30, (b)
no "Thanks for watching" in output, (c) raw WAV playable after
force-kill.
---
## Known unknowns / risks
- **`silero-vad` crate choice.** There's no first-party Rust binding;
the pragmatic options are `voice_activity_detector` (ort-backed,
pulls `ort` which is large) or porting Handy's direct `ort` call.
Decision point in Phase A.3; if the dep cost is unacceptable we fall
back to the existing RMS gate with stricter thresholds and document
the gap. Either way the `VadChunker` trait is the same, so the
downstream `StreamingSession` code doesn't change.
- **macOS App Nap** (#9): we cannot CI this on Linux. Ships with the
code pattern from Whispering's source and a manual-test-only note.
- **Windows installer Vulkan bundling** (#12): same — ships with
`tauri.conf.json` changes + a README note, verified manually on next
Windows build.
- **`build.rs` compile-time assert** (#6): if `cargo metadata` reports
`tokenizers` in any tree position on Windows, we `println!("cargo:
warning=...")` + `panic!` out of the build. Easier to read than a
runtime crash on first model load.
---
## Commit conventions on this branch
- One concern per commit. Subject line `feat(A.2):`, `fix(A.1):`, etc.,
referencing the phase so the reviewer can group them.
- Commit message body states which brief-item the commit closes.
- `cargo test --workspace --lib && cargo build -p kon && npm run check`
green before every commit, per the rules of engagement.