Lumotia/docs/whisper-ecosystem/workstream-A.md

Workstream A — Systems hardening + streaming correctness

Execution plan for the Codex half of the Whisper-ecosystem pass. Branch: phase4-systems-f7d0 (base: main). Companion: phase4-ux-f7d0 (Workstream B).

This is the backend / systems half. Every task below maps to an atomic item in docs/whisper-ecosystem/brief.md. Items #3, #4 (UX side only), #5, #10, #11, #14 (UI), #15, #16, #17, #20, #27, #30, #31 are owned by Workstream B and are listed here only where the contract is shared.

---

Scope recap

In scope for A: items #1, #2, #6, #7, #8, #9, #12, #13, #18, #19, #21, #22, #23, #24, #25, #26, #28, #29.

Explicit skip: #18 (initial-prompt priming) is already shipped — src-tauri/src/commands/mod.rs::build_initial_prompt plus crates/transcription/src/whisper_rs_backend.rs already wire the request prompt + profile prompt + profile vocabulary into FullParams::set_initial_prompt. Workstream A verifies via the existing lib tests and moves on; no new code.

Untouchable (owned by B): src/lib/pages/*.svelte, src/routes/**, crates/ai-formatting/src/llm_client.rs (B may only edit CLEANUP_PROMPT on that file).

Untouchable (global): the ggml dedup workaround in src-tauri/build.rs is pinned as a known interim — do not touch.

---

Execution order

The backlog groups itself into four natural phases. Each phase lands as a sequence of small, one-concern commits, with cargo test --workspace --lib && cargo build -p magnotia && npm run check green at every commit boundary. At the end of each phase we pause for review before the next.

Phase A.1 — Pre-emptive patches (no new UX surface)

Seq	Item	Effect	Depends on
1	#2: pre-approve http://127.0.0.1:* in Tauri capabilities for LLM	Adds http:default-scope-127.0.0.1 style capability + widens connect-src CSP to localhost	—
2	#6: guard whisper-rs-sys + tokenizers on Windows	Audit: Magnotia doesn't link tokenizers directly; we add a #[cfg(all(target_os = "windows"))] compile-time assert in magnotia-transcription/build.rs that fails the build if anything in the tree pulls in tokenizers on Windows	—
3	#7: CPU feature detection + non-AVX2 fallback surface	Extend crates/core/src/hardware.rs with CpuFeatures (avx2, avx512, fma, sse4_2) via std::is_x86_feature_detected! at runtime; surface as part of RuntimeCapabilities; when AVX2 is absent, emit a warning log + runtime-warning event so the frontend can show a banner	#1
4	#1: detect and surface active compute device	Extend RuntimeCapabilities with an ActiveComputeDevice struct (kind: "cuda" | "vulkan" | "metal" | "cpu", label: String, reason: Option<String>). For Whisper, read the first whisper_print_system_info line via a shim in whisper_rs_backend; reason is filled on any CPU fallback (e.g., "Vulkan loader not found")	#7
5	#8: checksum + resumable model downloads; retain audio on failure	Port the crates/llm/src/model_manager.rs pattern (SHA256 verify, .part file resume, ResumeUnsupported error) into crates/transcription/src/model_manager.rs::download_file. The existing code already does incremental SHA256 and range resume but lacks (a) validation of an existing full file before re-downloading, (b) ResumeUnsupported signalling, (c) test coverage parity. Retaining audio on failure is already covered in commands/live.rs via save_audio — add an explicit retry-friendly error classification so the frontend can render "Retry transcription" (a B surface, but the error enum ships here)	—
6	#9: disable macOS App Nap during capture + LLM	Add commands/power.rs with cfg(target_os = "macos") using NSProcessInfo beginActivityWithOptions:reason: to pin a power-assertion handle during live sessions and LLM generation	—
7	#12: bundle Vulkan loader + libssl on Windows, CPU fallback	Update src-tauri/tauri.conf.json bundle resources to list vulkan-1.dll and libssl-3-x64.dll/libcrypto-3-x64.dll; add a cfg(target_os = "windows") startup hook that probes LoadLibraryW("vulkan-1.dll") and downgrades active_compute_device to CPU with a clear reason if absent. Ship a pre-cargo build note in CI (README) — we cannot sign the installer from Linux CI	#1

Commit boundary: cargo test --workspace --lib (must include new tests for checksum resume + CPU feature detect + active-device shim), cargo build -p magnotia, npm run check. Then stop for review.

Phase A.2 — Engine abstraction (#13, #19)

Seq	Item	Effect	Depends on
8	#13 (engine trait): magnotia-transcription already has LocalEngine and a SpeechBackend enum over transcribe-rs adapter + WhisperRsBackend. This is a 90% version of Handy's transcribe-rs pattern. We lift it one more inch: replace the enum with a public Transcriber trait (load, transcribe_sync, capabilities) + blanket impls for the existing two backends, and gate WhisperRsBackend behind the whisper feature so non-Whisper builds compile without pulling whisper-rs-sys	—
9	#19: progressive WAV write during live capture	Extend commands/live.rs to stream captured mono-16kHz samples directly into a .wav in the session folder via magnotia_audio::WavWriter::append. Replace the in-memory kept_audio: Vec<f32> when save_audio is on with a disk-backed writer; on crash, the partial file is a playable WAV. Commits the commands/live.rs frame-size bookkeeping untouched	#13 (uses Transcriber::capabilities().sample_rate)

Commit boundary: tests (transcriber_trait_is_object_safe, wav_writer_survives_crash), build, check. Stop for review.

Phase A.3 — Streaming correctness (#21–#26)

This is the biggest change surface. We replace the custom RMS speech gate in commands/live.rs with a Silero-VAD-gated chunker and layer hallucination defences on top. Everything goes into a new crates/transcription/src/streaming/ module; commands/live.rs becomes a thin driver.

Seq	Item	Effect	Depends on
10	#23: warm-up silent WAV at app launch and after 60s idle	Add a tiny 1s silent-WAV loader (shipped in src-tauri/assets/warmup.wav) that runs through LocalEngine::transcribe_sync once after load_model; reschedules itself after any 60s idle gap in live sessions. Fixes the 4–5s cold-start latency cliff	#13
11	#21: Silero-VAD-gated chunker	Add silero-vad (ort-backed, small ONNX model) as a dependency. New streaming::VadChunker takes a 16 kHz mono stream, emits (chunk_start_sample, samples) pairs when the VAD score crosses a threshold with hysteresis. Configurable threshold (default 0.5, hysteresis 0.35). Replaces the RMS-based evaluate_speech_gate for the chunk-boundary decision. Current evaluate_speech_gate stays as a second-line silence skip	#19 (buffer ownership moved)
12	#22: hallucination blocklist + avg_logprob/no_speech_prob gate	Extend WhisperRsBackend to surface no_speech_prob + avg_logprob per segment (currently discarded). Add streaming::HallucinationFilter with a blocklist (static + user-editable via profile terms — blocked list, not vocab) and a confidence gate. Drop whole chunks that score below gate; drop segments whose text matches blocklist phrases	#21
13	#24: LocalAgreement-n commit policy	Add streaming::CommitPolicy::LocalAgreement { n: 2 }. Two consecutive passes must produce matching prefix tokens before emission; tentative tail is sent with a tentative: true flag on LiveResultMessage.segments for the UI (B renders the dashed underline — contract below)	#22
14	#25: aggressive buffer trim tied to commit points	Replace the current "drain OVERLAP_SAMPLES" with "drain everything up to the last commit point emitted by CommitPolicy". Guarantees the working buffer stays bounded regardless of wall-clock session length	#24
15	#26: repetition detector + context window reset	Add streaming::RepetitionDetector: three consecutive identical token runs (per whisper-rs token, or per-word if running Parakeet) triggers WhisperContext::create_state + drop the offending chunk, logs a runtime-warning event	#22

Commit boundary: tests for the VAD chunker (with a fixture WAV that contains silence → speech → silence), the LocalAgreement commit policy (feed two fake passes, assert only the agreed prefix emits), and the repetition detector. Stop for review.

Phase A.4 — LLM guard (#28, #29)

Seq	Item	Effect	Depends on
16	#28: sequential Whisper→LLM execution; single-GPU guard	Add crates/transcription/src/concurrency.rs::GpuGuard (already exists as file; extend it) — an async Semaphore::new(1) gate that wraps every transcribe_sync and every LlmEngine::generate on single-GPU systems. A parallel_mode: bool setting (default false; exposed via get_runtime_capabilities for B to wire into Settings) opens the gate to n = 2 for users with ≥16 GB VRAM detected by probe_gpu	#1
17	#29: plain-text pre-formatter before LLM prompt	crates/ai-formatting/src/pipeline.rs already joins segments; extend it to strip timestamps/IDs and normalise whitespace for the LLM call. New crates/ai-formatting/src/to_plain_text.rs with a pure function + tests	—

Commit boundary: tests, build, check. Stop for review.

---

Functions extended vs replaced

Existing function / module	Extend or replace	Rationale
commands::mod::build_initial_prompt	Keep as-is	Already covers #18
commands::models::get_runtime_capabilities	Extend	Add active_compute_device, cpu_features, parallel_mode_available fields
commands::live::evaluate_speech_gate	Replace with VAD-gated chunker, keep RMS version as a cheap fallback when VAD ONNX model missing
commands::live::run_live_session	Extend	Switch buffer ownership to streaming::StreamingSession, keep the cpal plumbing
crates/transcription/src/model_manager::download_file	Extend	Harden to match magnotia-llm's resume + sha behaviour, add ResumeUnsupported arm
crates/transcription/src/whisper_rs_backend::transcribe_sync	Extend	Surface no_speech_prob + avg_logprob per segment
crates/transcription/src/local_engine::SpeechBackend	Replace with public Transcriber trait
crates/transcription/src/concurrency	Extend	Already a stub; add GPU guard wrapper
crates/ai-formatting/src/pipeline	Extend only the plain-text join path
crates/core/src/hardware::probe_system	Extend	Add cpu_features + flesh out probe_gpu (which currently returns None) with a best-effort Vulkan loader probe

---

New commands and event contracts (for Workstream B)

Workstream B needs stable surfaces to wire into. These are the only Rust-side contracts A commits to in Phase A.1 and A.2 (Phase A.3/A.4 contracts are listed inline above):

Item #1: active compute device

Extend RuntimeCapabilities (no new command):

{
  "accelerators": ["cpu", "vulkan"],
  "activeComputeDevice": {
    "kind": "vulkan",                  // "cuda" | "vulkan" | "metal" | "cpu"
    "label": "NVIDIA GeForce RTX 3060 (Vulkan)",
    "reason": null                      // or "Vulkan loader not found, running on CPU"
  },
  "cpuFeatures": {
    "avx2": true,
    "avx512": false,
    "fma": true,
    "sse4_2": true
  },
  "parallelModeAvailable": false,       // true only when >= 16 GB VRAM
  "engines": [ /* unchanged */ ]
}

Event:

event: runtime-warning
payload: { kind: "avx2-missing" | "vulkan-loader-missing" | "cuda-fallback", message: string }

Emitted once at startup; B shows a dismissible Settings banner.

Item #14: GPU enumeration (for B's explicit selector)

New command, for B to wire the Settings dropdown:

command: list_gpus()
returns: [
  {
    "id": "cuda:0",
    "label": "NVIDIA GeForce RTX 3060",
    "vram_mb": 12288,
    "backend": "cuda",                 // "cuda" | "vulkan" | "metal" | "cpu"
    "active": true
  },
  {
    "id": "cpu",
    "label": "CPU (Ryzen 7 7840U, 16 threads)",
    "vram_mb": null,
    "backend": "cpu",
    "active": false
  }
]

command: set_preferred_gpu(id: string)
returns: Ok(()) — persists to settings table, takes effect on next model load

Note that actually swapping the device requires rebuilding the whisper-cpp backend with a different feature flag; for now this command stores the preference, surfaces a "Restart to apply" toast, and prewarm_default_model re-checks it on next launch.

Item #19 / #23: warm-up and progressive WAV

No new frontend-visible command. The save_audio flag on start_live_transcription_session already covers this; B should continue to use it.

Item #24: tentative vs committed segments (streaming)

LiveResultMessage.segments[i] gets a new field:

{
  "start": 1.2,
  "end": 1.8,
  "text": "hello world",
  "tentative": false   // true when emitted under LocalAgreement tail
}

B is expected to render tentative: true with a dashed underline in the preview overlay. Existing consumers that ignore unknown fields keep working; this field is additive.

Item #28: parallel-mode setting

Extend SettingsState on B's side to include parallelWhisperLlm: boolean (default false). B wires this to a new Rust command:

command: set_parallel_gpu_mode(enabled: bool) -> Ok(())

which flips the GpuGuard's semaphore permit count. Gated in Settings behind runtimeCapabilities.parallelModeAvailable.

---

Test strategy

• Every phase ends green on cargo test --workspace --lib && cargo build -p magnotia && npm run check.
• phase4-systems-f7d0 never regresses the 116 existing lib tests. If a test starts failing and the cause is an A change, it's fixed in-phase, not deferred.
• New unit tests land with their phase: the VAD chunker has a WAV fixture test, the LocalAgreement policy has a synthetic two-pass test, the download resume already has a tokio::net::TcpListener fixture test — we port the pattern.
• Manual smoke test after Phase A.3 on Jake's dev box: 10-minute live dictation session, assert (a) no latency growth past chunk 30, (b) no "Thanks for watching" in output, (c) raw WAV playable after force-kill.

---

Known unknowns / risks

• silero-vad crate choice. There's no first-party Rust binding; the pragmatic options are voice_activity_detector (ort-backed, pulls ort which is large) or porting Handy's direct ort call. Decision point in Phase A.3; if the dep cost is unacceptable we fall back to the existing RMS gate with stricter thresholds and document the gap. Either way the VadChunker trait is the same, so the downstream StreamingSession code doesn't change.
• macOS App Nap (#9): we cannot CI this on Linux. Ships with the code pattern from Whispering's source and a manual-test-only note.
• Windows installer Vulkan bundling (#12): same — ships with tauri.conf.json changes + a README note, verified manually on next Windows build.
• build.rs compile-time assert (#6): if cargo metadata reports tokenizers in any tree position on Windows, we println!("cargo: warning=...") + panic! out of the build. Easier to read than a runtime crash on first model load.

---

Commit conventions on this branch

• One concern per commit. Subject line feat(A.2):, fix(A.1):, etc., referencing the phase so the reviewer can group them.
• Commit message body states which brief-item the commit closes.
• cargo test --workspace --lib && cargo build -p magnotia && npm run check green before every commit, per the rules of engagement.