docs(gpu-tuning): add MVP plan — three phases with one-click UX

Specs the subset of the five-phase GPU kernel tuning roadmap that ships without requiring ggml-dedup or agentic-search prerequisites: - Phase 1 — Advanced → GPU Tuning settings panel (GGML env var toggles, applied at startup before threads spawn). - Phase 2 — kon-bench local autotuning CLI. Subprocess-based grid search over env vars, outputs a ranked gpu-profile.toml. - Phase 3-lite — kon-configs community repo. Manual-PR workflow (no CI replay), fingerprint-matched fetch from Kon Settings. Total ~7–10 days of focused work; captures roughly 85% of the eventual value of the full roadmap. Phases 4–5 (custom SPIR-V drops + agentic autotune) stay pinned in memory. Includes the UX spec for the "one-click auto-optimise" flow: community config check first (~15 s end-to-end), local benchmark fallback (~8 min backgrounded), opt-in share-back via browser PR. Non-GPU users see a clean "tuning doesn't apply" card with no nag. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 12:32:43 +01:00
parent ff22497468
commit 65280c776e
1 changed files with 364 additions and 0 deletions
--- a/docs/gpu-tuning/plan.md
+++ b/docs/gpu-tuning/plan.md
@@ -0,0 +1,364 @@
+# Kon — GPU Tuning & Community Config Plan
+
+*Implementation spec for the first three phases of the GPU kernel tuning roadmap. The full five-phase roadmap is pinned in memory; this document scopes the MVP subset that ships real value without pulling in `ggml`-dedup or agentic-search prerequisites.*
+
+## Scope
+
+**IN** (this document):
+
+- Phase 1 — Advanced GPU tuning settings panel (exposing GGML env vars)
+- Phase 2 — `kon-bench` local autotuning CLI
+- Phase 3-lite — `kon-configs` community repo with manual-PR workflow (no CI replay)
+
+**OUT** (pinned to memory for later):
+
+- Phase 4 — custom SPIR-V shader drops (blocked on `ggml`-dedup)
+- Phase 5 — Karpathy-style agentic autotuning
+- CI replay for community repo (defer until spam / bad configs become a real problem)
+
+This subset captures roughly 85% of the perceived value for ~20% of the total effort. The deferred pieces are where complexity explodes; the MVP stops before it.
+
+---
+
+## Phase 1 — Advanced GPU tuning settings panel
+
+**Effort**: 1–2 days.
+
+**What ships**: Settings → Advanced → GPU Tuning collapsible section with toggles for GGML env vars. Env vars are applied at app startup before any GPU backend initialises. Per-profile storage; restart required to take effect.
+
+### Toggles shipped at MVP
+
+| UI label | Env var | Default | When users enable |
+|---|---|---|---|
+| Disable cooperative matrix | `GGML_VK_DISABLE_COOPMAT` | off | "Inference hangs" on RDNA2 / buggy Mesa versions |
+| Force FP32 math | `GGML_VK_FORCE_FP32` | off | "Garbage transcripts" on Intel Arc / older NVIDIA |
+| Disable FP16 ops | `GGML_VK_DISABLE_F16` | off | Silent-fail on some Mesa 22.x builds |
+| Disable integer dot product | `GGML_VK_DISABLE_INTEGER_DOT_PRODUCT` | off | "Random NaN" on RDNA2 with certain drivers |
+| Enable Vulkan validation | `GGML_VK_VALIDATE` | off | Diagnostic only; impacts performance |
+
+Metal / CUDA counterparts slot in when those backends grow in Kon. Today Kon is Vulkan-only.
+
+### Design
+
+- New `SettingsState.gpuTuning: { disableCoopmat: boolean, forceFp32: boolean, disableF16: boolean, disableIntegerDotProduct: boolean, enableValidation: boolean }` in [src/lib/types/app.ts](../../src/lib/types/app.ts)
+- All defaults `false` in [src/lib/stores/page.svelte.ts](../../src/lib/stores/page.svelte.ts)
+- Persistence uses the existing `save_preferences` → SQLite `kon_preferences` path
+- Backend reads preferences at the **very top** of `run()` in [src-tauri/src/lib.rs](../../src-tauri/src/lib.rs) — before `tauri::Builder::default()` spawns threads — and writes via `unsafe { std::env::set_var(...) }`. Matches the existing `ensure_x11_on_wayland` pattern
+- Settings UI shows a sticky "Restart required for changes to take effect" banner when any toggle has drifted from its launch-time value
+- A "Reset to defaults" button zeroes all toggles
+
+### Acceptance
+
+- Toggling "Disable cooperative matrix" on and restarting → `vulkaninfo` (or GGML debug logs) confirms the knob is honoured at backend init
+- Default all-off produces identical performance + transcription output to the current `main` (smoke test)
+- An integration test with a fake settings fixture confirms env vars are set before `AppState` initialises
+
+---
+
+## Phase 2 — `kon-bench` local autotuning CLI
+
+**Effort**: 3–5 days.
+
+**What ships**: New workspace binary `crates/bench/` producing a `kon-bench` executable. User runs it once post-install; output lands at `~/.kon/gpu-profile.toml` with the best-scoring config for their hardware. Settings page gets an "Apply auto-tuned profile" button that consumes the TOML and updates the Phase 1 toggles.
+
+### CLI surface
+
+```
+kon-bench --quick                      # bundled 20s sample + reference transcript
+kon-bench --model <path> --audio <wav> --transcript <txt>
+kon-bench --compare <profile.toml>     # benchmark a specific profile vs default
+```
+
+### Execution model
+
+Grid-search via **subprocess spawning**. Each config variant runs in a child process with its own env vars — because env vars must be set at process startup; you cannot safely mutate GGML's runtime state once it's initialised. The parent serialises variants, spawns a child per variant, waits for each to exit with a JSON line on stdout, aggregates and ranks.
+
+### Search strategy (not naive combinatorial)
+
+1. Run baseline (all defaults).
+2. Run each single-flag variant against baseline.
+3. Take the top-3 single flags by RTF improvement with zero WER drift.
+4. Combine pairwise.
+5. Top-scored composite config wins.
+
+This gives us ~9–15 subprocess runs instead of the ~32 a full combinatorial sweep would need; converges on local optima without the combinatorial explosion.
+
+### Metrics
+
+- **Real-time factor (RTF)** = `audio_seconds / inference_wall_seconds`. Lower is better.
+- **Word error rate (WER)** against the ground-truth transcript. Any config with >0.5% WER drift from baseline is rejected regardless of RTF improvement.
+- **Peak VRAM** (optional, best-effort via `nvidia-smi` / `rocm-smi` sampling).
+
+### Runtime
+
+~5–15 minutes on typical hardware. Progress bar + ETA rendered to stderr so stdout stays machine-readable.
+
+### Bundled fixture
+
+A 20-second public-domain speech clip with a known-good reference transcript, committed to `crates/bench/fixtures/`. Source: LibriVox recording (CC0).
+
+### Output schema (`gpu-profile.toml`)
+
+```toml
+[benchmarked_at]
+timestamp = "2026-04-21T14:32:00Z"
+kon_version = "0.1.0"
+model = "whisper-distil-large-v3"
+
+[hardware]
+gpu_name = "NVIDIA GeForce RTX 4070"
+vram_mb = 12282
+driver = "nvidia 550.120"
+os = "linux"
+mesa = ""
+
+[baseline]
+rtf = 0.043
+wer = 0.028
+
+[best]
+rtf = 0.031
+rtf_improvement = 0.279    # 27.9% faster
+wer = 0.028
+
+[best.env]
+GGML_VK_DISABLE_COOPMAT = "0"
+GGML_VK_FORCE_FP32 = "0"
+# … full flag set, including unchanged ones, for reproducibility
+```
+
+### Crate layout
+
+```
+crates/bench/
+├── Cargo.toml
+├── fixtures/
+│   ├── librivox-sample.wav
+│   └── librivox-sample.txt
+└── src/
+    ├── main.rs         # CLI + parent process
+    ├── runner.rs       # subprocess harness (child entry gate: KON_BENCH_RUN=1)
+    ├── matrix.rs       # grid-search + top-k logic
+    ├── metrics.rs      # RTF + WER + optional VRAM sampling
+    └── profile.rs      # TOML serialise
+```
+
+Depends on `kon-transcription` + `kon-llm` + `kon-audio` as path deps so it reuses the existing model-loading code.
+
+### Acceptance
+
+- `kon-bench --quick` runs unattended to completion on a fresh install
+- Produces a valid `gpu-profile.toml`
+- "Apply auto-tuned" button in Settings consumes the TOML and updates Phase 1 toggles (restart banner fires as expected)
+- Re-running with `--compare <profile>` produces reproducible-enough numbers (RTF within 5% run-to-run)
+
+---
+
+## Phase 3-lite — `kon-configs` community repo
+
+**Effort**: 3 days (1 for repo + seeds, 2 for Kon-side fetch + apply UI).
+
+**What ships**: A separate public GitHub repo `kon-configs` (not part of the kon main repo) seeded with 2–3 curated configs. Kon's Settings page gets a "Browse community configs" button that fetches matching configs for the user's detected hardware.
+
+### Repo structure
+
+```
+kon-configs/
+├── README.md              # pitch + how to benefit
+├── CONTRIBUTING.md        # required fields, benchmark protocol, fork/PR flow
+├── SCHEMA.md              # TOML schema documentation
+├── index.json             # manifest for Kon to discover configs
+└── configs/
+    ├── nvidia/
+    │   ├── rtx-3060-12gb-linux.toml
+    │   └── rtx-4070-linux.toml
+    ├── amd/
+    │   └── rx-6700xt-mesa-23-linux.toml
+    └── intel/
+        └── arc-a770-windows.toml
+```
+
+### Config TOML
+
+Extends Phase 2's `gpu-profile.toml` schema with an `[attribution]` section:
+
+```toml
+[attribution]
+submitter = "@username"
+notes = "Tested with 1-hour continuous dictation session, no crashes."
+```
+
+### Contribution flow (manual, honour-system MVP)
+
+1. User runs `kon-bench` on their hardware.
+2. User runs `kon-bench --compare` against baseline to confirm improvement isn't noise.
+3. User forks `kon-configs`, commits their TOML under `configs/<vendor>/`, opens PR.
+4. Maintainer reviews format + plausibility, merges.
+5. No CI replay — revisit if spam becomes a problem.
+
+### Kon integration
+
+- New Tauri command `fetch_community_configs(gpu_fingerprint)` — HTTPS GET `https://raw.githubusercontent.com/<org>/kon-configs/main/index.json` for the manifest, then fetches matching TOMLs
+- Fingerprint match: GPU name substring + VRAM tier (e.g., `"RTX 3060"` + `"12gb"`)
+- Settings "Browse community configs" button lists matches with submitter, claimed RTF improvement, and a preview of the toggle deltas
+- Applying a config updates Phase 1 toggles AND stores provenance (source = `"community"`, submitter, fetch date)
+
+### What we explicitly skip at MVP
+
+- **No CI replay**. Maintainer eyeballs + honour system. Revisit past ~50 configs or on abuse.
+- **No automated upload from `kon-bench`**. User always commits + PRs manually. Zero privacy concerns, zero spam surface.
+- **No sophisticated fingerprint normalisation**. Substring matching is sufficient.
+
+### Acceptance
+
+- Repo exists with README + CONTRIBUTING + 2–3 seed configs
+- Kon Settings fetches + lists + applies a community config end-to-end
+- "Revert to default" path works (Phase 1's reset)
+
+---
+
+## User experience — the one-click path
+
+This is the UX the three phases together enable. All three are prerequisites; Phase 3-lite is what turns "run a CLI" into "click a button."
+
+### First-launch onboarding nudge
+
+After the existing first-run model download, Kon surfaces a non-modal card:
+
+```
+🎛  GPU Optimisation
+Detected: NVIDIA RTX 4070 (12 GB) · Linux Wayland
+Current: Default GGML kernels
+
+[ Auto-optimise ]   [ Show advanced ]   [ Skip ]
+```
+
+"Auto-optimise" triggers the hybrid flow below. "Show advanced" expands the Phase 1 toggle panel directly. "Skip" dismisses; user can always come back via Settings.
+
+### The "Auto-optimise" flow
+
+Two steps, in this order:
+
+**Step 1 — Community config check (instant, ~2 s)**
+
+Kon fingerprints the GPU and queries the `kon-configs` manifest for matches. If a match exists, a preview card appears:
+
+```
+┌─────────────────────────────────────────────┐
+│ Community config available                  │
+│                                             │
+│ From:     @someuser                         │
+│ Claimed:  27% faster · 0% accuracy drift    │
+│ Tested:   2026-04-21, driver nvidia 550     │
+│                                             │
+│ Changes 2 settings:                         │
+│   • Cooperative matrix:  on → off           │
+│   • Integer dot product: on → off           │
+│                                             │
+│ [ Apply (restart required) ]   [ Cancel ]   │
+└─────────────────────────────────────────────┘
+```
+
+Apply → settings persist → restart prompt → done. 15 seconds end-to-end.
+
+**Step 2 — Fallback to local benchmark**
+
+If no community match, or the user prefers their own measurement:
+
+```
+┌─────────────────────────────────────────────┐
+│ No community config for your hardware yet   │
+│                                             │
+│ We can benchmark your machine to find the   │
+│ best settings. Takes ~8 minutes; runs in    │
+│ the background while you keep using Kon.    │
+│                                             │
+│ [ Benchmark my GPU ]   [ Skip ]             │
+└─────────────────────────────────────────────┘
+```
+
+Kicks off `kon-bench` as a background process. Kon keeps working during the run.
+
+### Progress UI during benchmark
+
+Non-modal. Status chip in the lower-right of the main window:
+
+```
+⚙  Benchmarking GPU · 4 of 12 tested · ~5 min remaining   [ cancel ]
+```
+
+On completion, a toast:
+
+```
+Your GPU is 27% faster with the new config.   [ Review → ]
+```
+
+Review opens the same preview card as the community-config flow, with the same Apply / Cancel options.
+
+### After applying
+
+Settings shows the active config's provenance:
+
+- `Using community config · applied 2026-04-21 · by @someuser`
+- `Using auto-tuned config · benchmarked 2026-04-21`
+- `Using defaults`
+
+Plus a "Revert to previous config" button, active for 7 days after any change, in case the new config misbehaves in real use (silent accuracy drift, crashes on long sessions, etc.) that the benchmark didn't catch.
+
+### Optional — sharing back to the community
+
+After a successful local benchmark that shows meaningful gains, Kon prompts:
+
+```
+┌─────────────────────────────────────────────┐
+│ Share your config with the community?       │
+│                                             │
+│ Your RTX 4070 tuning got you 27% faster.    │
+│ Other RTX 4070 users would benefit.         │
+│                                             │
+│ Shared data: GPU name, driver version, OS,  │
+│ config flags, benchmark numbers.            │
+│ NOT shared: personal info, audio, anything  │
+│ that identifies you beyond the GitHub fork. │
+│                                             │
+│ [ Review payload ]   [ Create PR ]   [ No ] │
+└─────────────────────────────────────────────┘
+```
+
+"Create PR" opens the user's browser to `github.com/…/kon-configs/new/main` with the TOML prefilled in the PR body. User finishes the submission on GitHub (still honour-system; no automated uploads, no telemetry).
+
+### Non-GPU / integrated-only fallback
+
+If `sysinfo` reports no dedicated GPU or Vulkan isn't available, the card replaces itself with:
+
+```
+🎛  GPU Optimisation
+No dedicated GPU detected — Kon is using CPU inference.
+GPU tuning doesn't apply to this setup.
+```
+
+No nag, no hidden settings, no broken experience.
+
+### Yes, "one click" is achievable
+
+For users whose GPU has a community-contributed config, the experience is **literally one click** (the Apply button), plus a restart. ~15 seconds.
+
+For users without a community match, the experience is **two clicks** (trigger bench → apply results on completion), with a passive ~8-minute background wait in between.
+
+For users on integrated graphics / no GPU, the experience is **zero clicks** — Kon tells them GPU tuning doesn't apply and moves on.
+
+---
+
+## Sequencing
+
+Strict linear: Phase 1 → Phase 2 → Phase 3-lite. Each phase merges to `main` and gets dogfooded before the next starts.
+
+- Phase 1 is a prereq for Phase 2 — `kon-bench`'s output needs the Phase 1 settings schema to be its consumption target.
+- Phase 2 is a prereq for Phase 3-lite — the community repo's config TOML schema **is** Phase 2's output schema (with an added `[attribution]` section).
+
+## Shelved with rationale
+
+- **Phase 4 — custom SPIR-V shader drops.** Blocked on `ggml`-dedup workstream. Pinned in memory.
+- **Phase 5 — agentic (Karpathy-style) autotune.** Phase 2's grid search produces schema-compatible results, so Phase 5 can drop in later without a schema break. Pinned.
+- **Phase 3's CI replay.** Defer until spam / bad-config abuse is a real problem rather than a hypothetical one. Honour-system PR review is sufficient for the MVP community.
+- **`kon-bench` automated upload.** Deliberately manual for MVP — removes all privacy / spam / rate-limiting concerns. Revisit when the community volume justifies the infrastructure.