From 35cfdfddf19f3a5e38cb25aa94fbe77958774473 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Tue, 21 Apr 2026 10:55:18 +0000 Subject: [PATCH] docs(phase4-systems): execution plan for Workstream A Sequences the 18 A-scope items from docs/whisper-ecosystem/brief.md into four phases (pre-emptive patches, engine abstraction, streaming correctness, LLM guard) with stop-for-review boundaries between each. Lists the new command + event contracts (#1 activeComputeDevice, #14 list_gpus/set_preferred_gpu, #24 tentative segment flag, #28 parallel-mode toggle) that Workstream B will consume. Co-authored-by: jars --- docs/whisper-ecosystem/workstream-A.md | 270 +++++++++++++++++++++++++ 1 file changed, 270 insertions(+) create mode 100644 docs/whisper-ecosystem/workstream-A.md diff --git a/docs/whisper-ecosystem/workstream-A.md b/docs/whisper-ecosystem/workstream-A.md new file mode 100644 index 0000000..e4201b6 --- /dev/null +++ b/docs/whisper-ecosystem/workstream-A.md @@ -0,0 +1,270 @@ +# Workstream A — Systems hardening + streaming correctness + +*Execution plan for the Codex half of the Whisper-ecosystem pass.* +*Branch: `phase4-systems-f7d0` (base: `main`). Companion: `phase4-ux-f7d0` (Workstream B).* + +This is the backend / systems half. Every task below maps to an atomic +item in `docs/whisper-ecosystem/brief.md`. Items #3, #4 (UX side only), +#5, #10, #11, #14 (UI), #15, #16, #17, #20, #27, #30, #31 are owned by +Workstream B and are listed here only where the contract is shared. + +--- + +## Scope recap + +**In scope for A:** items #1, #2, #6, #7, #8, #9, #12, #13, #18, #19, +#21, #22, #23, #24, #25, #26, #28, #29. + +**Explicit skip:** #18 (initial-prompt priming) is already shipped — +`src-tauri/src/commands/mod.rs::build_initial_prompt` plus +`crates/transcription/src/whisper_rs_backend.rs` already wire the +request prompt + profile prompt + profile vocabulary into +`FullParams::set_initial_prompt`. Workstream A verifies via the existing +lib tests and moves on; no new code. + +**Untouchable (owned by B):** `src/lib/pages/*.svelte`, `src/routes/**`, +`crates/ai-formatting/src/llm_client.rs` (B may only edit +`CLEANUP_PROMPT` on that file). + +**Untouchable (global):** the `ggml` dedup workaround in +`src-tauri/build.rs` is pinned as a known interim — do not touch. + +--- + +## Execution order + +The backlog groups itself into four natural phases. Each phase lands as +a sequence of small, one-concern commits, with `cargo test --workspace +--lib && cargo build -p kon && npm run check` green at every commit +boundary. At the end of each phase we pause for review before the next. + +### Phase A.1 — Pre-emptive patches (no new UX surface) + +| Seq | Item | Effect | Depends on | +|---|---|---|---| +| 1 | **#2**: pre-approve `http://127.0.0.1:*` in Tauri capabilities for LLM | Adds `http:default-scope-127.0.0.1` style capability + widens `connect-src` CSP to localhost | — | +| 2 | **#6**: guard `whisper-rs-sys` + `tokenizers` on Windows | Audit: Kon doesn't link `tokenizers` directly; we add a `#[cfg(all(target_os = "windows"))]` compile-time assert in `kon-transcription/build.rs` that fails the build if anything in the tree pulls in `tokenizers` on Windows | — | +| 3 | **#7**: CPU feature detection + non-AVX2 fallback surface | Extend `crates/core/src/hardware.rs` with `CpuFeatures` (avx2, avx512, fma, sse4_2) via `std::is_x86_feature_detected!` at runtime; surface as part of `RuntimeCapabilities`; when AVX2 is absent, emit a `warning` log + `runtime-warning` event so the frontend can show a banner | #1 | +| 4 | **#1**: detect and surface active compute device | Extend `RuntimeCapabilities` with an `ActiveComputeDevice` struct (`kind: "cuda" \| "vulkan" \| "metal" \| "cpu"`, `label: String`, `reason: Option`). For Whisper, read the first `whisper_print_system_info` line via a shim in `whisper_rs_backend`; reason is filled on any CPU fallback (e.g., "Vulkan loader not found") | #7 | +| 5 | **#8**: checksum + resumable model downloads; retain audio on failure | Port the `crates/llm/src/model_manager.rs` pattern (SHA256 verify, `.part` file resume, `ResumeUnsupported` error) into `crates/transcription/src/model_manager.rs::download_file`. The existing code already does incremental SHA256 and range resume but lacks (a) validation of an **existing full file** before re-downloading, (b) ResumeUnsupported signalling, (c) test coverage parity. Retaining audio on failure is already covered in `commands/live.rs` via `save_audio` — add an explicit retry-friendly error classification so the frontend can render "Retry transcription" (a B surface, but the error enum ships here) | — | +| 6 | **#9**: disable macOS App Nap during capture + LLM | Add `commands/power.rs` with `cfg(target_os = "macos")` using `NSProcessInfo beginActivityWithOptions:reason:` to pin a power-assertion handle during live sessions and LLM generation | — | +| 7 | **#12**: bundle Vulkan loader + libssl on Windows, CPU fallback | Update `src-tauri/tauri.conf.json` bundle `resources` to list `vulkan-1.dll` and `libssl-3-x64.dll`/`libcrypto-3-x64.dll`; add a `cfg(target_os = "windows")` startup hook that probes `LoadLibraryW("vulkan-1.dll")` and downgrades `active_compute_device` to CPU with a clear reason if absent. Ship a pre-`cargo build` note in CI (README) — we cannot sign the installer from Linux CI | #1 | + +**Commit boundary:** `cargo test --workspace --lib` (must include new +tests for checksum resume + CPU feature detect + active-device shim), +`cargo build -p kon`, `npm run check`. Then **stop for review.** + +### Phase A.2 — Engine abstraction (#13, #19) + +| Seq | Item | Effect | Depends on | +|---|---|---|---| +| 8 | **#13 (engine trait)**: `kon-transcription` already has `LocalEngine` and a `SpeechBackend` enum over `transcribe-rs` adapter + `WhisperRsBackend`. This is a 90% version of Handy's `transcribe-rs` pattern. We lift it one more inch: replace the enum with a public `Transcriber` trait (`load`, `transcribe_sync`, `capabilities`) + blanket impls for the existing two backends, and gate `WhisperRsBackend` behind the `whisper` feature so non-Whisper builds compile without pulling `whisper-rs-sys` | — | +| 9 | **#19**: progressive WAV write during live capture | Extend `commands/live.rs` to stream captured mono-16kHz samples directly into a `.wav` in the session folder via `kon_audio::WavWriter::append`. Replace the in-memory `kept_audio: Vec` when `save_audio` is on with a disk-backed writer; on crash, the partial file is a playable WAV. Commits the `commands/live.rs` frame-size bookkeeping untouched | #13 (uses `Transcriber::capabilities().sample_rate`) | + +**Commit boundary:** tests (`transcriber_trait_is_object_safe`, +`wav_writer_survives_crash`), build, check. **Stop for review.** + +### Phase A.3 — Streaming correctness (#21–#26) + +This is the biggest change surface. We replace the custom RMS speech +gate in `commands/live.rs` with a Silero-VAD-gated chunker and layer +hallucination defences on top. Everything goes into a new +`crates/transcription/src/streaming/` module; `commands/live.rs` +becomes a thin driver. + +| Seq | Item | Effect | Depends on | +|---|---|---|---| +| 10 | **#23**: warm-up silent WAV at app launch and after 60s idle | Add a tiny 1s silent-WAV loader (shipped in `src-tauri/assets/warmup.wav`) that runs through `LocalEngine::transcribe_sync` once after load_model; reschedules itself after any 60s idle gap in live sessions. Fixes the 4–5s cold-start latency cliff | #13 | +| 11 | **#21**: Silero-VAD-gated chunker | Add `silero-vad` (ort-backed, small ONNX model) as a dependency. New `streaming::VadChunker` takes a 16 kHz mono stream, emits `(chunk_start_sample, samples)` pairs when the VAD score crosses a threshold with hysteresis. Configurable threshold (default 0.5, hysteresis 0.35). Replaces the RMS-based `evaluate_speech_gate` for the chunk-boundary decision. Current `evaluate_speech_gate` stays as a _second-line_ silence skip | #19 (buffer ownership moved) | +| 12 | **#22**: hallucination blocklist + `avg_logprob`/`no_speech_prob` gate | Extend `WhisperRsBackend` to surface `no_speech_prob` + `avg_logprob` per segment (currently discarded). Add `streaming::HallucinationFilter` with a blocklist (static + user-editable via profile terms — blocked list, not vocab) and a confidence gate. Drop whole chunks that score below gate; drop segments whose text matches blocklist phrases | #21 | +| 13 | **#24**: LocalAgreement-n commit policy | Add `streaming::CommitPolicy::LocalAgreement { n: 2 }`. Two consecutive passes must produce matching prefix tokens before emission; tentative tail is sent with a `tentative: true` flag on `LiveResultMessage.segments` for the UI (B renders the dashed underline — contract below) | #22 | +| 14 | **#25**: aggressive buffer trim tied to commit points | Replace the current "drain OVERLAP_SAMPLES" with "drain everything up to the last commit point emitted by `CommitPolicy`". Guarantees the working buffer stays bounded regardless of wall-clock session length | #24 | +| 15 | **#26**: repetition detector + context window reset | Add `streaming::RepetitionDetector`: three consecutive identical token runs (per whisper-rs token, or per-word if running Parakeet) triggers `WhisperContext::create_state` + drop the offending chunk, logs a `runtime-warning` event | #22 | + +**Commit boundary:** tests for the VAD chunker (with a fixture WAV that +contains silence → speech → silence), the LocalAgreement commit policy +(feed two fake passes, assert only the agreed prefix emits), and the +repetition detector. **Stop for review.** + +### Phase A.4 — LLM guard (#28, #29) + +| Seq | Item | Effect | Depends on | +|---|---|---|---| +| 16 | **#28**: sequential Whisper→LLM execution; single-GPU guard | Add `crates/transcription/src/concurrency.rs::GpuGuard` (already exists as file; extend it) — an async `Semaphore::new(1)` gate that wraps every `transcribe_sync` and every `LlmEngine::generate` on single-GPU systems. A `parallel_mode: bool` setting (default false; exposed via `get_runtime_capabilities` for B to wire into Settings) opens the gate to `n = 2` for users with ≥16 GB VRAM detected by `probe_gpu` | #1 | +| 17 | **#29**: plain-text pre-formatter before LLM prompt | `crates/ai-formatting/src/pipeline.rs` already joins segments; extend it to strip timestamps/IDs and normalise whitespace for the LLM call. New `crates/ai-formatting/src/to_plain_text.rs` with a pure function + tests | — | + +**Commit boundary:** tests, build, check. **Stop for review.** + +--- + +## Functions extended vs replaced + +| Existing function / module | Extend or replace | Rationale | +|---|---|---| +| `commands::mod::build_initial_prompt` | **Keep as-is** | Already covers #18 | +| `commands::models::get_runtime_capabilities` | **Extend** | Add `active_compute_device`, `cpu_features`, `parallel_mode_available` fields | +| `commands::live::evaluate_speech_gate` | **Replace** with VAD-gated chunker, keep RMS version as a cheap fallback when VAD ONNX model missing | +| `commands::live::run_live_session` | **Extend** | Switch buffer ownership to `streaming::StreamingSession`, keep the `cpal` plumbing | +| `crates/transcription/src/model_manager::download_file` | **Extend** | Harden to match `kon-llm`'s resume + sha behaviour, add `ResumeUnsupported` arm | +| `crates/transcription/src/whisper_rs_backend::transcribe_sync` | **Extend** | Surface `no_speech_prob` + `avg_logprob` per segment | +| `crates/transcription/src/local_engine::SpeechBackend` | **Replace** with public `Transcriber` trait | +| `crates/transcription/src/concurrency` | **Extend** | Already a stub; add GPU guard wrapper | +| `crates/ai-formatting/src/pipeline` | **Extend** only the plain-text join path | +| `crates/core/src/hardware::probe_system` | **Extend** | Add `cpu_features` + flesh out `probe_gpu` (which currently returns `None`) with a best-effort Vulkan loader probe | + +--- + +## New commands and event contracts (for Workstream B) + +Workstream B needs stable surfaces to wire into. These are the only +Rust-side contracts A commits to in Phase A.1 and A.2 (Phase A.3/A.4 +contracts are listed inline above): + +### Item #1: active compute device + +Extend `RuntimeCapabilities` (no new command): + +```jsonc +{ + "accelerators": ["cpu", "vulkan"], + "activeComputeDevice": { + "kind": "vulkan", // "cuda" | "vulkan" | "metal" | "cpu" + "label": "NVIDIA GeForce RTX 3060 (Vulkan)", + "reason": null // or "Vulkan loader not found, running on CPU" + }, + "cpuFeatures": { + "avx2": true, + "avx512": false, + "fma": true, + "sse4_2": true + }, + "parallelModeAvailable": false, // true only when >= 16 GB VRAM + "engines": [ /* unchanged */ ] +} +``` + +Event: + +``` +event: runtime-warning +payload: { kind: "avx2-missing" | "vulkan-loader-missing" | "cuda-fallback", message: string } +``` + +Emitted once at startup; B shows a dismissible Settings banner. + +### Item #14: GPU enumeration (for B's explicit selector) + +New command, for B to wire the Settings dropdown: + +``` +command: list_gpus() +returns: [ + { + "id": "cuda:0", + "label": "NVIDIA GeForce RTX 3060", + "vram_mb": 12288, + "backend": "cuda", // "cuda" | "vulkan" | "metal" | "cpu" + "active": true + }, + { + "id": "cpu", + "label": "CPU (Ryzen 7 7840U, 16 threads)", + "vram_mb": null, + "backend": "cpu", + "active": false + } +] + +command: set_preferred_gpu(id: string) +returns: Ok(()) — persists to settings table, takes effect on next model load +``` + +Note that **actually swapping** the device requires rebuilding the +whisper-cpp backend with a different feature flag; for now this command +stores the preference, surfaces a "Restart to apply" toast, and +`prewarm_default_model` re-checks it on next launch. + +### Item #19 / #23: warm-up and progressive WAV + +No new frontend-visible command. The `save_audio` flag on +`start_live_transcription_session` already covers this; B should +continue to use it. + +### Item #24: tentative vs committed segments (streaming) + +`LiveResultMessage.segments[i]` gets a new field: + +``` +{ + "start": 1.2, + "end": 1.8, + "text": "hello world", + "tentative": false // true when emitted under LocalAgreement tail +} +``` + +B is expected to render `tentative: true` with a dashed underline in +the preview overlay. Existing consumers that ignore unknown fields keep +working; this field is additive. + +### Item #28: parallel-mode setting + +Extend `SettingsState` on B's side to include +`parallelWhisperLlm: boolean` (default `false`). B wires this to a new +Rust command: + +``` +command: set_parallel_gpu_mode(enabled: bool) -> Ok(()) +``` + +which flips the `GpuGuard`'s semaphore permit count. Gated in Settings +behind `runtimeCapabilities.parallelModeAvailable`. + +--- + +## Test strategy + +- Every phase ends green on `cargo test --workspace --lib && cargo + build -p kon && npm run check`. +- `phase4-systems-f7d0` never regresses the 116 existing lib tests. If + a test starts failing and the cause is an A change, it's fixed + in-phase, not deferred. +- New unit tests land with their phase: the VAD chunker has a WAV + fixture test, the LocalAgreement policy has a synthetic two-pass + test, the download resume already has a `tokio::net::TcpListener` + fixture test — we port the pattern. +- Manual smoke test after Phase A.3 on Jake's dev box: 10-minute live + dictation session, assert (a) no latency growth past chunk 30, (b) + no "Thanks for watching" in output, (c) raw WAV playable after + force-kill. + +--- + +## Known unknowns / risks + +- **`silero-vad` crate choice.** There's no first-party Rust binding; + the pragmatic options are `voice_activity_detector` (ort-backed, + pulls `ort` which is large) or porting Handy's direct `ort` call. + Decision point in Phase A.3; if the dep cost is unacceptable we fall + back to the existing RMS gate with stricter thresholds and document + the gap. Either way the `VadChunker` trait is the same, so the + downstream `StreamingSession` code doesn't change. +- **macOS App Nap** (#9): we cannot CI this on Linux. Ships with the + code pattern from Whispering's source and a manual-test-only note. +- **Windows installer Vulkan bundling** (#12): same — ships with + `tauri.conf.json` changes + a README note, verified manually on next + Windows build. +- **`build.rs` compile-time assert** (#6): if `cargo metadata` reports + `tokenizers` in any tree position on Windows, we `println!("cargo: + warning=...")` + `panic!` out of the build. Easier to read than a + runtime crash on first model load. + +--- + +## Commit conventions on this branch + +- One concern per commit. Subject line `feat(A.2):`, `fix(A.1):`, etc., + referencing the phase so the reviewer can group them. +- Commit message body states which brief-item the commit closes. +- `cargo test --workspace --lib && cargo build -p kon && npm run check` + green before every commit, per the rules of engagement.