Lumotia/docs/whisper-ecosystem/workstream-A.md

# Workstream A — Systems hardening + streaming correctness

*Execution plan for the Codex half of the Whisper-ecosystem pass.*
*Branch: `phase4-systems-f7d0` (base: `main`). Companion: `phase4-ux-f7d0` (Workstream B).*

This is the backend / systems half. Every task below maps to an atomic
item in `docs/whisper-ecosystem/brief.md`. Items #3, #4 (UX side only),
#5, #10, #11, #14 (UI), #15, #16, #17, #20, #27, #30, #31 are owned by
Workstream B and are listed here only where the contract is shared.

---

## Scope recap

**In scope for A:** items #1, #2, #6, #7, #8, #9, #12, #13, #18, #19,
#21, #22, #23, #24, #25, #26, #28, #29.

**Explicit skip:** #18 (initial-prompt priming) is already shipped —
`src-tauri/src/commands/mod.rs::build_initial_prompt` plus
`crates/transcription/src/whisper_rs_backend.rs` already wire the
request prompt + profile prompt + profile vocabulary into
`FullParams::set_initial_prompt`. Workstream A verifies via the existing
lib tests and moves on; no new code.

**Untouchable (owned by B):** `src/lib/pages/*.svelte`, `src/routes/**`,
`crates/ai-formatting/src/llm_client.rs` (B may only edit
`CLEANUP_PROMPT` on that file).

**Untouchable (global):** the `ggml` dedup workaround in
`src-tauri/build.rs` is pinned as a known interim — do not touch.

---

## Execution order

The backlog groups itself into four natural phases. Each phase lands as
a sequence of small, one-concern commits, with `cargo test --workspace
--lib && cargo build -p lumotia && npm run check` green at every commit
boundary. At the end of each phase we pause for review before the next.

### Phase A.1 — Pre-emptive patches (no new UX surface)

| Seq | Item | Effect | Depends on |
|---|---|---|---|
| 1 | **#2**: pre-approve `http://127.0.0.1:*` in Tauri capabilities for LLM | Adds `http:default-scope-127.0.0.1` style capability + widens `connect-src` CSP to localhost | — |
| 2 | **#6**: guard `whisper-rs-sys` + `tokenizers` on Windows | Audit: Lumotia doesn't link `tokenizers` directly; we add a `#[cfg(all(target_os = "windows"))]` compile-time assert in `lumotia-transcription/build.rs` that fails the build if anything in the tree pulls in `tokenizers` on Windows | — |
| 3 | **#7**: CPU feature detection + non-AVX2 fallback surface | Extend `crates/core/src/hardware.rs` with `CpuFeatures` (avx2, avx512, fma, sse4_2) via `std::is_x86_feature_detected!` at runtime; surface as part of `RuntimeCapabilities`; when AVX2 is absent, emit a `warning` log + `runtime-warning` event so the frontend can show a banner | #1 |
| 4 | **#1**: detect and surface active compute device | Extend `RuntimeCapabilities` with an `ActiveComputeDevice` struct (`kind: "cuda" \| "vulkan" \| "metal" \| "cpu"`, `label: String`, `reason: Option<String>`). For Whisper, read the first `whisper_print_system_info` line via a shim in `whisper_rs_backend`; reason is filled on any CPU fallback (e.g., "Vulkan loader not found") | #7 |
| 5 | **#8**: checksum + resumable model downloads; retain audio on failure | Port the `crates/llm/src/model_manager.rs` pattern (SHA256 verify, `.part` file resume, `ResumeUnsupported` error) into `crates/transcription/src/model_manager.rs::download_file`. The existing code already does incremental SHA256 and range resume but lacks (a) validation of an **existing full file** before re-downloading, (b) ResumeUnsupported signalling, (c) test coverage parity. Retaining audio on failure is already covered in `commands/live.rs` via `save_audio` — add an explicit retry-friendly error classification so the frontend can render "Retry transcription" (a B surface, but the error enum ships here) | — |
| 6 | **#9**: disable macOS App Nap during capture + LLM | Add `commands/power.rs` with `cfg(target_os = "macos")` using `NSProcessInfo beginActivityWithOptions:reason:` to pin a power-assertion handle during live sessions and LLM generation | — |
| 7 | **#12**: bundle Vulkan loader + libssl on Windows, CPU fallback | Update `src-tauri/tauri.conf.json` bundle `resources` to list `vulkan-1.dll` and `libssl-3-x64.dll`/`libcrypto-3-x64.dll`; add a `cfg(target_os = "windows")` startup hook that probes `LoadLibraryW("vulkan-1.dll")` and downgrades `active_compute_device` to CPU with a clear reason if absent. Ship a pre-`cargo build` note in CI (README) — we cannot sign the installer from Linux CI | #1 |

**Commit boundary:** `cargo test --workspace --lib` (must include new
tests for checksum resume + CPU feature detect + active-device shim),
`cargo build -p lumotia`, `npm run check`. Then **stop for review.**

### Phase A.2 — Engine abstraction (#13, #19)

| Seq | Item | Effect | Depends on |
|---|---|---|---|
| 8 | **#13 (engine trait)**: `lumotia-transcription` already has `LocalEngine` and a `SpeechBackend` enum over `transcribe-rs` adapter + `WhisperRsBackend`. This is a 90% version of Handy's `transcribe-rs` pattern. We lift it one more inch: replace the enum with a public `Transcriber` trait (`load`, `transcribe_sync`, `capabilities`) + blanket impls for the existing two backends, and gate `WhisperRsBackend` behind the `whisper` feature so non-Whisper builds compile without pulling `whisper-rs-sys` | — |
| 9 | **#19**: progressive WAV write during live capture | Extend `commands/live.rs` to stream captured mono-16kHz samples directly into a `.wav` in the session folder via `lumotia_audio::WavWriter::append`. Replace the in-memory `kept_audio: Vec<f32>` when `save_audio` is on with a disk-backed writer; on crash, the partial file is a playable WAV. Commits the `commands/live.rs` frame-size bookkeeping untouched | #13 (uses `Transcriber::capabilities().sample_rate`) |

**Commit boundary:** tests (`transcriber_trait_is_object_safe`,
`wav_writer_survives_crash`), build, check. **Stop for review.**

### Phase A.3 — Streaming correctness (#21–#26)

This is the biggest change surface. We replace the custom RMS speech
gate in `commands/live.rs` with a Silero-VAD-gated chunker and layer
hallucination defences on top. Everything goes into a new
`crates/transcription/src/streaming/` module; `commands/live.rs`
becomes a thin driver.

| Seq | Item | Effect | Depends on |
|---|---|---|---|
| 10 | **#23**: warm-up silent WAV at app launch and after 60s idle | Add a tiny 1s silent-WAV loader (shipped in `src-tauri/assets/warmup.wav`) that runs through `LocalEngine::transcribe_sync` once after load_model; reschedules itself after any 60s idle gap in live sessions. Fixes the 4–5s cold-start latency cliff | #13 |
| 11 | **#21**: Silero-VAD-gated chunker | Add `silero-vad` (ort-backed, small ONNX model) as a dependency. New `streaming::VadChunker` takes a 16 kHz mono stream, emits `(chunk_start_sample, samples)` pairs when the VAD score crosses a threshold with hysteresis. Configurable threshold (default 0.5, hysteresis 0.35). Replaces the RMS-based `evaluate_speech_gate` for the chunk-boundary decision. Current `evaluate_speech_gate` stays as a _second-line_ silence skip | #19 (buffer ownership moved) |
| 12 | **#22**: hallucination blocklist + `avg_logprob`/`no_speech_prob` gate | Extend `WhisperRsBackend` to surface `no_speech_prob` + `avg_logprob` per segment (currently discarded). Add `streaming::HallucinationFilter` with a blocklist (static + user-editable via profile terms — blocked list, not vocab) and a confidence gate. Drop whole chunks that score below gate; drop segments whose text matches blocklist phrases | #21 |
| 13 | **#24**: LocalAgreement-n commit policy | Add `streaming::CommitPolicy::LocalAgreement { n: 2 }`. Two consecutive passes must produce matching prefix tokens before emission; tentative tail is sent with a `tentative: true` flag on `LiveResultMessage.segments` for the UI (B renders the dashed underline — contract below) | #22 |
| 14 | **#25**: aggressive buffer trim tied to commit points | Replace the current "drain OVERLAP_SAMPLES" with "drain everything up to the last commit point emitted by `CommitPolicy`". Guarantees the working buffer stays bounded regardless of wall-clock session length | #24 |
| 15 | **#26**: repetition detector + context window reset | Add `streaming::RepetitionDetector`: three consecutive identical token runs (per whisper-rs token, or per-word if running Parakeet) triggers `WhisperContext::create_state` + drop the offending chunk, logs a `runtime-warning` event | #22 |

**Commit boundary:** tests for the VAD chunker (with a fixture WAV that
contains silence → speech → silence), the LocalAgreement commit policy
(feed two fake passes, assert only the agreed prefix emits), and the
repetition detector. **Stop for review.**

### Phase A.4 — LLM guard (#28, #29)

| Seq | Item | Effect | Depends on |
|---|---|---|---|
| 16 | **#28**: sequential Whisper→LLM execution; single-GPU guard | Add `crates/transcription/src/concurrency.rs::GpuGuard` (already exists as file; extend it) — an async `Semaphore::new(1)` gate that wraps every `transcribe_sync` and every `LlmEngine::generate` on single-GPU systems. A `parallel_mode: bool` setting (default false; exposed via `get_runtime_capabilities` for B to wire into Settings) opens the gate to `n = 2` for users with ≥16 GB VRAM detected by `probe_gpu` | #1 |
| 17 | **#29**: plain-text pre-formatter before LLM prompt | `crates/ai-formatting/src/pipeline.rs` already joins segments; extend it to strip timestamps/IDs and normalise whitespace for the LLM call. New `crates/ai-formatting/src/to_plain_text.rs` with a pure function + tests | — |

**Commit boundary:** tests, build, check. **Stop for review.**

---

## Functions extended vs replaced

| Existing function / module | Extend or replace | Rationale |
|---|---|---|
| `commands::mod::build_initial_prompt` | **Keep as-is** | Already covers #18 |
| `commands::models::get_runtime_capabilities` | **Extend** | Add `active_compute_device`, `cpu_features`, `parallel_mode_available` fields |
| `commands::live::evaluate_speech_gate` | **Replace** with VAD-gated chunker, keep RMS version as a cheap fallback when VAD ONNX model missing |
| `commands::live::run_live_session` | **Extend** | Switch buffer ownership to `streaming::StreamingSession`, keep the `cpal` plumbing |
| `crates/transcription/src/model_manager::download_file` | **Extend** | Harden to match `lumotia-llm`'s resume + sha behaviour, add `ResumeUnsupported` arm |
| `crates/transcription/src/whisper_rs_backend::transcribe_sync` | **Extend** | Surface `no_speech_prob` + `avg_logprob` per segment |
| `crates/transcription/src/local_engine::SpeechBackend` | **Replace** with public `Transcriber` trait |
| `crates/transcription/src/concurrency` | **Extend** | Already a stub; add GPU guard wrapper |
| `crates/ai-formatting/src/pipeline` | **Extend** only the plain-text join path |
| `crates/core/src/hardware::probe_system` | **Extend** | Add `cpu_features` + flesh out `probe_gpu` (which currently returns `None`) with a best-effort Vulkan loader probe |

---

## New commands and event contracts (for Workstream B)

Workstream B needs stable surfaces to wire into. These are the only
Rust-side contracts A commits to in Phase A.1 and A.2 (Phase A.3/A.4
contracts are listed inline above):

### Item #1: active compute device

Extend `RuntimeCapabilities` (no new command):

```jsonc
{
  "accelerators": ["cpu", "vulkan"],
  "activeComputeDevice": {
    "kind": "vulkan",                  // "cuda" | "vulkan" | "metal" | "cpu"
    "label": "NVIDIA GeForce RTX 3060 (Vulkan)",
    "reason": null                      // or "Vulkan loader not found, running on CPU"
  },
  "cpuFeatures": {
    "avx2": true,
    "avx512": false,
    "fma": true,
    "sse4_2": true
  },
  "parallelModeAvailable": false,       // true only when >= 16 GB VRAM
  "engines": [ /* unchanged */ ]
}
```

Event:

```
event: runtime-warning
payload: { kind: "avx2-missing" | "vulkan-loader-missing" | "cuda-fallback", message: string }
```

Emitted once at startup; B shows a dismissible Settings banner.

### Item #14: GPU enumeration (for B's explicit selector)

New command, for B to wire the Settings dropdown:

```
command: list_gpus()
returns: [
  {
    "id": "cuda:0",
    "label": "NVIDIA GeForce RTX 3060",
    "vram_mb": 12288,
    "backend": "cuda",                 // "cuda" | "vulkan" | "metal" | "cpu"
    "active": true
  },
  {
    "id": "cpu",
    "label": "CPU (Ryzen 7 7840U, 16 threads)",
    "vram_mb": null,
    "backend": "cpu",
    "active": false
  }
]

command: set_preferred_gpu(id: string)
returns: Ok(()) — persists to settings table, takes effect on next model load
```

Note that **actually swapping** the device requires rebuilding the
whisper-cpp backend with a different feature flag; for now this command
stores the preference, surfaces a "Restart to apply" toast, and
`prewarm_default_model` re-checks it on next launch.

### Item #19 / #23: warm-up and progressive WAV

No new frontend-visible command. The `save_audio` flag on
`start_live_transcription_session` already covers this; B should
continue to use it.

### Item #24: tentative vs committed segments (streaming)

`LiveResultMessage.segments[i]` gets a new field:

```
{
  "start": 1.2,
  "end": 1.8,
  "text": "hello world",
  "tentative": false   // true when emitted under LocalAgreement tail
}
```

B is expected to render `tentative: true` with a dashed underline in
the preview overlay. Existing consumers that ignore unknown fields keep
working; this field is additive.

### Item #28: parallel-mode setting

Extend `SettingsState` on B's side to include
`parallelWhisperLlm: boolean` (default `false`). B wires this to a new
Rust command:

```
command: set_parallel_gpu_mode(enabled: bool) -> Ok(())
```

which flips the `GpuGuard`'s semaphore permit count. Gated in Settings
behind `runtimeCapabilities.parallelModeAvailable`.

---

## Test strategy

- Every phase ends green on `cargo test --workspace --lib && cargo
  build -p lumotia && npm run check`.
- `phase4-systems-f7d0` never regresses the 116 existing lib tests. If
  a test starts failing and the cause is an A change, it's fixed
  in-phase, not deferred.
- New unit tests land with their phase: the VAD chunker has a WAV
  fixture test, the LocalAgreement policy has a synthetic two-pass
  test, the download resume already has a `tokio::net::TcpListener`
  fixture test — we port the pattern.
- Manual smoke test after Phase A.3 on Jake's dev box: 10-minute live
  dictation session, assert (a) no latency growth past chunk 30, (b)
  no "Thanks for watching" in output, (c) raw WAV playable after
  force-kill.

---

## Known unknowns / risks

- **`silero-vad` crate choice.** There's no first-party Rust binding;
  the pragmatic options are `voice_activity_detector` (ort-backed,
  pulls `ort` which is large) or porting Handy's direct `ort` call.
  Decision point in Phase A.3; if the dep cost is unacceptable we fall
  back to the existing RMS gate with stricter thresholds and document
  the gap. Either way the `VadChunker` trait is the same, so the
  downstream `StreamingSession` code doesn't change.
- **macOS App Nap** (#9): we cannot CI this on Linux. Ships with the
  code pattern from Whispering's source and a manual-test-only note.
- **Windows installer Vulkan bundling** (#12): same — ships with
  `tauri.conf.json` changes + a README note, verified manually on next
  Windows build.
- **`build.rs` compile-time assert** (#6): if `cargo metadata` reports
  `tokenizers` in any tree position on Windows, we `println!("cargo:
  warning=...")` + `panic!` out of the build. Easier to read than a
  runtime crash on first model load.

---

## Commit conventions on this branch

- One concern per commit. Subject line `feat(A.2):`, `fix(A.1):`, etc.,
  referencing the phase so the reviewer can group them.
- Commit message body states which brief-item the commit closes.
- `cargo test --workspace --lib && cargo build -p lumotia && npm run check`
  green before every commit, per the rules of engagement.