docs(phase4-systems): execution plan for Workstream A

Sequences the 18 A-scope items from docs/whisper-ecosystem/brief.md into four phases (pre-emptive patches, engine abstraction, streaming correctness, LLM guard) with stop-for-review boundaries between each. Lists the new command + event contracts (#1 activeComputeDevice, #14 list_gpus/set_preferred_gpu, #24 tentative segment flag, #28 parallel-mode toggle) that Workstream B will consume. Co-authored-by: jars <jakejars@users.noreply.github.com>
2026-04-21 10:55:18 +00:00
parent da74a84009
commit 35cfdfddf1
1 changed files with 270 additions and 0 deletions
--- a/docs/whisper-ecosystem/workstream-A.md
+++ b/docs/whisper-ecosystem/workstream-A.md
@@ -0,0 +1,270 @@
+# Workstream A — Systems hardening + streaming correctness
+
+*Execution plan for the Codex half of the Whisper-ecosystem pass.*
+*Branch: `phase4-systems-f7d0` (base: `main`). Companion: `phase4-ux-f7d0` (Workstream B).*
+
+This is the backend / systems half. Every task below maps to an atomic
+item in `docs/whisper-ecosystem/brief.md`. Items #3, #4 (UX side only),
+#5, #10, #11, #14 (UI), #15, #16, #17, #20, #27, #30, #31 are owned by
+Workstream B and are listed here only where the contract is shared.
+
+---
+
+## Scope recap
+
+**In scope for A:** items #1, #2, #6, #7, #8, #9, #12, #13, #18, #19,
+#21, #22, #23, #24, #25, #26, #28, #29.
+
+**Explicit skip:** #18 (initial-prompt priming) is already shipped —
+`src-tauri/src/commands/mod.rs::build_initial_prompt` plus
+`crates/transcription/src/whisper_rs_backend.rs` already wire the
+request prompt + profile prompt + profile vocabulary into
+`FullParams::set_initial_prompt`. Workstream A verifies via the existing
+lib tests and moves on; no new code.
+
+**Untouchable (owned by B):** `src/lib/pages/*.svelte`, `src/routes/**`,
+`crates/ai-formatting/src/llm_client.rs` (B may only edit
+`CLEANUP_PROMPT` on that file).
+
+**Untouchable (global):** the `ggml` dedup workaround in
+`src-tauri/build.rs` is pinned as a known interim — do not touch.
+
+---
+
+## Execution order
+
+The backlog groups itself into four natural phases. Each phase lands as
+a sequence of small, one-concern commits, with `cargo test --workspace
+--lib && cargo build -p kon && npm run check` green at every commit
+boundary. At the end of each phase we pause for review before the next.
+
+### Phase A.1 — Pre-emptive patches (no new UX surface)
+
+| Seq | Item | Effect | Depends on |
+|---|---|---|---|
+| 1 | **#2**: pre-approve `http://127.0.0.1:*` in Tauri capabilities for LLM | Adds `http:default-scope-127.0.0.1` style capability + widens `connect-src` CSP to localhost | — |
+| 2 | **#6**: guard `whisper-rs-sys` + `tokenizers` on Windows | Audit: Kon doesn't link `tokenizers` directly; we add a `#[cfg(all(target_os = "windows"))]` compile-time assert in `kon-transcription/build.rs` that fails the build if anything in the tree pulls in `tokenizers` on Windows | — |
+| 3 | **#7**: CPU feature detection + non-AVX2 fallback surface | Extend `crates/core/src/hardware.rs` with `CpuFeatures` (avx2, avx512, fma, sse4_2) via `std::is_x86_feature_detected!` at runtime; surface as part of `RuntimeCapabilities`; when AVX2 is absent, emit a `warning` log + `runtime-warning` event so the frontend can show a banner | #1 |
+| 4 | **#1**: detect and surface active compute device | Extend `RuntimeCapabilities` with an `ActiveComputeDevice` struct (`kind: "cuda" \| "vulkan" \| "metal" \| "cpu"`, `label: String`, `reason: Option<String>`). For Whisper, read the first `whisper_print_system_info` line via a shim in `whisper_rs_backend`; reason is filled on any CPU fallback (e.g., "Vulkan loader not found") | #7 |
+| 5 | **#8**: checksum + resumable model downloads; retain audio on failure | Port the `crates/llm/src/model_manager.rs` pattern (SHA256 verify, `.part` file resume, `ResumeUnsupported` error) into `crates/transcription/src/model_manager.rs::download_file`. The existing code already does incremental SHA256 and range resume but lacks (a) validation of an **existing full file** before re-downloading, (b) ResumeUnsupported signalling, (c) test coverage parity. Retaining audio on failure is already covered in `commands/live.rs` via `save_audio` — add an explicit retry-friendly error classification so the frontend can render "Retry transcription" (a B surface, but the error enum ships here) | — |
+| 6 | **#9**: disable macOS App Nap during capture + LLM | Add `commands/power.rs` with `cfg(target_os = "macos")` using `NSProcessInfo beginActivityWithOptions:reason:` to pin a power-assertion handle during live sessions and LLM generation | — |
+| 7 | **#12**: bundle Vulkan loader + libssl on Windows, CPU fallback | Update `src-tauri/tauri.conf.json` bundle `resources` to list `vulkan-1.dll` and `libssl-3-x64.dll`/`libcrypto-3-x64.dll`; add a `cfg(target_os = "windows")` startup hook that probes `LoadLibraryW("vulkan-1.dll")` and downgrades `active_compute_device` to CPU with a clear reason if absent. Ship a pre-`cargo build` note in CI (README) — we cannot sign the installer from Linux CI | #1 |
+
+**Commit boundary:** `cargo test --workspace --lib` (must include new
+tests for checksum resume + CPU feature detect + active-device shim),
+`cargo build -p kon`, `npm run check`. Then **stop for review.**
+
+### Phase A.2 — Engine abstraction (#13, #19)
+
+| Seq | Item | Effect | Depends on |
+|---|---|---|---|
+| 8 | **#13 (engine trait)**: `kon-transcription` already has `LocalEngine` and a `SpeechBackend` enum over `transcribe-rs` adapter + `WhisperRsBackend`. This is a 90% version of Handy's `transcribe-rs` pattern. We lift it one more inch: replace the enum with a public `Transcriber` trait (`load`, `transcribe_sync`, `capabilities`) + blanket impls for the existing two backends, and gate `WhisperRsBackend` behind the `whisper` feature so non-Whisper builds compile without pulling `whisper-rs-sys` | — |
+| 9 | **#19**: progressive WAV write during live capture | Extend `commands/live.rs` to stream captured mono-16kHz samples directly into a `.wav` in the session folder via `kon_audio::WavWriter::append`. Replace the in-memory `kept_audio: Vec<f32>` when `save_audio` is on with a disk-backed writer; on crash, the partial file is a playable WAV. Commits the `commands/live.rs` frame-size bookkeeping untouched | #13 (uses `Transcriber::capabilities().sample_rate`) |
+
+**Commit boundary:** tests (`transcriber_trait_is_object_safe`,
+`wav_writer_survives_crash`), build, check. **Stop for review.**
+
+### Phase A.3 — Streaming correctness (#21–#26)
+
+This is the biggest change surface. We replace the custom RMS speech
+gate in `commands/live.rs` with a Silero-VAD-gated chunker and layer
+hallucination defences on top. Everything goes into a new
+`crates/transcription/src/streaming/` module; `commands/live.rs`
+becomes a thin driver.
+
+| Seq | Item | Effect | Depends on |
+|---|---|---|---|
+| 10 | **#23**: warm-up silent WAV at app launch and after 60s idle | Add a tiny 1s silent-WAV loader (shipped in `src-tauri/assets/warmup.wav`) that runs through `LocalEngine::transcribe_sync` once after load_model; reschedules itself after any 60s idle gap in live sessions. Fixes the 4–5s cold-start latency cliff | #13 |
+| 11 | **#21**: Silero-VAD-gated chunker | Add `silero-vad` (ort-backed, small ONNX model) as a dependency. New `streaming::VadChunker` takes a 16 kHz mono stream, emits `(chunk_start_sample, samples)` pairs when the VAD score crosses a threshold with hysteresis. Configurable threshold (default 0.5, hysteresis 0.35). Replaces the RMS-based `evaluate_speech_gate` for the chunk-boundary decision. Current `evaluate_speech_gate` stays as a _second-line_ silence skip | #19 (buffer ownership moved) |
+| 12 | **#22**: hallucination blocklist + `avg_logprob`/`no_speech_prob` gate | Extend `WhisperRsBackend` to surface `no_speech_prob` + `avg_logprob` per segment (currently discarded). Add `streaming::HallucinationFilter` with a blocklist (static + user-editable via profile terms — blocked list, not vocab) and a confidence gate. Drop whole chunks that score below gate; drop segments whose text matches blocklist phrases | #21 |
+| 13 | **#24**: LocalAgreement-n commit policy | Add `streaming::CommitPolicy::LocalAgreement { n: 2 }`. Two consecutive passes must produce matching prefix tokens before emission; tentative tail is sent with a `tentative: true` flag on `LiveResultMessage.segments` for the UI (B renders the dashed underline — contract below) | #22 |
+| 14 | **#25**: aggressive buffer trim tied to commit points | Replace the current "drain OVERLAP_SAMPLES" with "drain everything up to the last commit point emitted by `CommitPolicy`". Guarantees the working buffer stays bounded regardless of wall-clock session length | #24 |
+| 15 | **#26**: repetition detector + context window reset | Add `streaming::RepetitionDetector`: three consecutive identical token runs (per whisper-rs token, or per-word if running Parakeet) triggers `WhisperContext::create_state` + drop the offending chunk, logs a `runtime-warning` event | #22 |
+
+**Commit boundary:** tests for the VAD chunker (with a fixture WAV that
+contains silence → speech → silence), the LocalAgreement commit policy
+(feed two fake passes, assert only the agreed prefix emits), and the
+repetition detector. **Stop for review.**
+
+### Phase A.4 — LLM guard (#28, #29)
+
+| Seq | Item | Effect | Depends on |
+|---|---|---|---|
+| 16 | **#28**: sequential Whisper→LLM execution; single-GPU guard | Add `crates/transcription/src/concurrency.rs::GpuGuard` (already exists as file; extend it) — an async `Semaphore::new(1)` gate that wraps every `transcribe_sync` and every `LlmEngine::generate` on single-GPU systems. A `parallel_mode: bool` setting (default false; exposed via `get_runtime_capabilities` for B to wire into Settings) opens the gate to `n = 2` for users with ≥16 GB VRAM detected by `probe_gpu` | #1 |
+| 17 | **#29**: plain-text pre-formatter before LLM prompt | `crates/ai-formatting/src/pipeline.rs` already joins segments; extend it to strip timestamps/IDs and normalise whitespace for the LLM call. New `crates/ai-formatting/src/to_plain_text.rs` with a pure function + tests | — |
+
+**Commit boundary:** tests, build, check. **Stop for review.**
+
+---
+
+## Functions extended vs replaced
+
+| Existing function / module | Extend or replace | Rationale |
+|---|---|---|
+| `commands::mod::build_initial_prompt` | **Keep as-is** | Already covers #18 |
+| `commands::models::get_runtime_capabilities` | **Extend** | Add `active_compute_device`, `cpu_features`, `parallel_mode_available` fields |
+| `commands::live::evaluate_speech_gate` | **Replace** with VAD-gated chunker, keep RMS version as a cheap fallback when VAD ONNX model missing |
+| `commands::live::run_live_session` | **Extend** | Switch buffer ownership to `streaming::StreamingSession`, keep the `cpal` plumbing |
+| `crates/transcription/src/model_manager::download_file` | **Extend** | Harden to match `kon-llm`'s resume + sha behaviour, add `ResumeUnsupported` arm |
+| `crates/transcription/src/whisper_rs_backend::transcribe_sync` | **Extend** | Surface `no_speech_prob` + `avg_logprob` per segment |
+| `crates/transcription/src/local_engine::SpeechBackend` | **Replace** with public `Transcriber` trait |
+| `crates/transcription/src/concurrency` | **Extend** | Already a stub; add GPU guard wrapper |
+| `crates/ai-formatting/src/pipeline` | **Extend** only the plain-text join path |
+| `crates/core/src/hardware::probe_system` | **Extend** | Add `cpu_features` + flesh out `probe_gpu` (which currently returns `None`) with a best-effort Vulkan loader probe |
+
+---
+
+## New commands and event contracts (for Workstream B)
+
+Workstream B needs stable surfaces to wire into. These are the only
+Rust-side contracts A commits to in Phase A.1 and A.2 (Phase A.3/A.4
+contracts are listed inline above):
+
+### Item #1: active compute device
+
+Extend `RuntimeCapabilities` (no new command):
+
+```jsonc
+{
+  "accelerators": ["cpu", "vulkan"],
+  "activeComputeDevice": {
+    "kind": "vulkan",                  // "cuda" | "vulkan" | "metal" | "cpu"
+    "label": "NVIDIA GeForce RTX 3060 (Vulkan)",
+    "reason": null                      // or "Vulkan loader not found, running on CPU"
+  },
+  "cpuFeatures": {
+    "avx2": true,
+    "avx512": false,
+    "fma": true,
+    "sse4_2": true
+  },
+  "parallelModeAvailable": false,       // true only when >= 16 GB VRAM
+  "engines": [ /* unchanged */ ]
+}
+```
+
+Event:
+
+```
+event: runtime-warning
+payload: { kind: "avx2-missing" | "vulkan-loader-missing" | "cuda-fallback", message: string }
+```
+
+Emitted once at startup; B shows a dismissible Settings banner.
+
+### Item #14: GPU enumeration (for B's explicit selector)
+
+New command, for B to wire the Settings dropdown:
+
+```
+command: list_gpus()
+returns: [
+  {
+    "id": "cuda:0",
+    "label": "NVIDIA GeForce RTX 3060",
+    "vram_mb": 12288,
+    "backend": "cuda",                 // "cuda" | "vulkan" | "metal" | "cpu"
+    "active": true
+  },
+  {
+    "id": "cpu",
+    "label": "CPU (Ryzen 7 7840U, 16 threads)",
+    "vram_mb": null,
+    "backend": "cpu",
+    "active": false
+  }
+]
+
+command: set_preferred_gpu(id: string)
+returns: Ok(()) — persists to settings table, takes effect on next model load
+```
+
+Note that **actually swapping** the device requires rebuilding the
+whisper-cpp backend with a different feature flag; for now this command
+stores the preference, surfaces a "Restart to apply" toast, and
+`prewarm_default_model` re-checks it on next launch.
+
+### Item #19 / #23: warm-up and progressive WAV
+
+No new frontend-visible command. The `save_audio` flag on
+`start_live_transcription_session` already covers this; B should
+continue to use it.
+
+### Item #24: tentative vs committed segments (streaming)
+
+`LiveResultMessage.segments[i]` gets a new field:
+
+```
+{
+  "start": 1.2,
+  "end": 1.8,
+  "text": "hello world",
+  "tentative": false   // true when emitted under LocalAgreement tail
+}
+```
+
+B is expected to render `tentative: true` with a dashed underline in
+the preview overlay. Existing consumers that ignore unknown fields keep
+working; this field is additive.
+
+### Item #28: parallel-mode setting
+
+Extend `SettingsState` on B's side to include
+`parallelWhisperLlm: boolean` (default `false`). B wires this to a new
+Rust command:
+
+```
+command: set_parallel_gpu_mode(enabled: bool) -> Ok(())
+```
+
+which flips the `GpuGuard`'s semaphore permit count. Gated in Settings
+behind `runtimeCapabilities.parallelModeAvailable`.
+
+---
+
+## Test strategy
+
+- Every phase ends green on `cargo test --workspace --lib && cargo
+  build -p kon && npm run check`.
+- `phase4-systems-f7d0` never regresses the 116 existing lib tests. If
+  a test starts failing and the cause is an A change, it's fixed
+  in-phase, not deferred.
+- New unit tests land with their phase: the VAD chunker has a WAV
+  fixture test, the LocalAgreement policy has a synthetic two-pass
+  test, the download resume already has a `tokio::net::TcpListener`
+  fixture test — we port the pattern.
+- Manual smoke test after Phase A.3 on Jake's dev box: 10-minute live
+  dictation session, assert (a) no latency growth past chunk 30, (b)
+  no "Thanks for watching" in output, (c) raw WAV playable after
+  force-kill.
+
+---
+
+## Known unknowns / risks
+
+- **`silero-vad` crate choice.** There's no first-party Rust binding;
+  the pragmatic options are `voice_activity_detector` (ort-backed,
+  pulls `ort` which is large) or porting Handy's direct `ort` call.
+  Decision point in Phase A.3; if the dep cost is unacceptable we fall
+  back to the existing RMS gate with stricter thresholds and document
+  the gap. Either way the `VadChunker` trait is the same, so the
+  downstream `StreamingSession` code doesn't change.
+- **macOS App Nap** (#9): we cannot CI this on Linux. Ships with the
+  code pattern from Whispering's source and a manual-test-only note.
+- **Windows installer Vulkan bundling** (#12): same — ships with
+  `tauri.conf.json` changes + a README note, verified manually on next
+  Windows build.
+- **`build.rs` compile-time assert** (#6): if `cargo metadata` reports
+  `tokenizers` in any tree position on Windows, we `println!("cargo:
+  warning=...")` + `panic!` out of the build. Easier to read than a
+  runtime crash on first model load.
+
+---
+
+## Commit conventions on this branch
+
+- One concern per commit. Subject line `feat(A.2):`, `fix(A.1):`, etc.,
+  referencing the phase so the reviewer can group them.
+- Commit message body states which brief-item the commit closes.
+- `cargo test --workspace --lib && cargo build -p kon && npm run check`
+  green before every commit, per the rules of engagement.