diff --git a/docs/gpu-tuning/plan.md b/docs/gpu-tuning/plan.md new file mode 100644 index 0000000..22369a4 --- /dev/null +++ b/docs/gpu-tuning/plan.md @@ -0,0 +1,364 @@ +# Kon — GPU Tuning & Community Config Plan + +*Implementation spec for the first three phases of the GPU kernel tuning roadmap. The full five-phase roadmap is pinned in memory; this document scopes the MVP subset that ships real value without pulling in `ggml`-dedup or agentic-search prerequisites.* + +## Scope + +**IN** (this document): + +- Phase 1 — Advanced GPU tuning settings panel (exposing GGML env vars) +- Phase 2 — `kon-bench` local autotuning CLI +- Phase 3-lite — `kon-configs` community repo with manual-PR workflow (no CI replay) + +**OUT** (pinned to memory for later): + +- Phase 4 — custom SPIR-V shader drops (blocked on `ggml`-dedup) +- Phase 5 — Karpathy-style agentic autotuning +- CI replay for community repo (defer until spam / bad configs become a real problem) + +This subset captures roughly 85% of the perceived value for ~20% of the total effort. The deferred pieces are where complexity explodes; the MVP stops before it. + +--- + +## Phase 1 — Advanced GPU tuning settings panel + +**Effort**: 1–2 days. + +**What ships**: Settings → Advanced → GPU Tuning collapsible section with toggles for GGML env vars. Env vars are applied at app startup before any GPU backend initialises. Per-profile storage; restart required to take effect. + +### Toggles shipped at MVP + +| UI label | Env var | Default | When users enable | +|---|---|---|---| +| Disable cooperative matrix | `GGML_VK_DISABLE_COOPMAT` | off | "Inference hangs" on RDNA2 / buggy Mesa versions | +| Force FP32 math | `GGML_VK_FORCE_FP32` | off | "Garbage transcripts" on Intel Arc / older NVIDIA | +| Disable FP16 ops | `GGML_VK_DISABLE_F16` | off | Silent-fail on some Mesa 22.x builds | +| Disable integer dot product | `GGML_VK_DISABLE_INTEGER_DOT_PRODUCT` | off | "Random NaN" on RDNA2 with certain drivers | +| Enable Vulkan validation | `GGML_VK_VALIDATE` | off | Diagnostic only; impacts performance | + +Metal / CUDA counterparts slot in when those backends grow in Kon. Today Kon is Vulkan-only. + +### Design + +- New `SettingsState.gpuTuning: { disableCoopmat: boolean, forceFp32: boolean, disableF16: boolean, disableIntegerDotProduct: boolean, enableValidation: boolean }` in [src/lib/types/app.ts](../../src/lib/types/app.ts) +- All defaults `false` in [src/lib/stores/page.svelte.ts](../../src/lib/stores/page.svelte.ts) +- Persistence uses the existing `save_preferences` → SQLite `kon_preferences` path +- Backend reads preferences at the **very top** of `run()` in [src-tauri/src/lib.rs](../../src-tauri/src/lib.rs) — before `tauri::Builder::default()` spawns threads — and writes via `unsafe { std::env::set_var(...) }`. Matches the existing `ensure_x11_on_wayland` pattern +- Settings UI shows a sticky "Restart required for changes to take effect" banner when any toggle has drifted from its launch-time value +- A "Reset to defaults" button zeroes all toggles + +### Acceptance + +- Toggling "Disable cooperative matrix" on and restarting → `vulkaninfo` (or GGML debug logs) confirms the knob is honoured at backend init +- Default all-off produces identical performance + transcription output to the current `main` (smoke test) +- An integration test with a fake settings fixture confirms env vars are set before `AppState` initialises + +--- + +## Phase 2 — `kon-bench` local autotuning CLI + +**Effort**: 3–5 days. + +**What ships**: New workspace binary `crates/bench/` producing a `kon-bench` executable. User runs it once post-install; output lands at `~/.kon/gpu-profile.toml` with the best-scoring config for their hardware. Settings page gets an "Apply auto-tuned profile" button that consumes the TOML and updates the Phase 1 toggles. + +### CLI surface + +``` +kon-bench --quick # bundled 20s sample + reference transcript +kon-bench --model --audio --transcript +kon-bench --compare # benchmark a specific profile vs default +``` + +### Execution model + +Grid-search via **subprocess spawning**. Each config variant runs in a child process with its own env vars — because env vars must be set at process startup; you cannot safely mutate GGML's runtime state once it's initialised. The parent serialises variants, spawns a child per variant, waits for each to exit with a JSON line on stdout, aggregates and ranks. + +### Search strategy (not naive combinatorial) + +1. Run baseline (all defaults). +2. Run each single-flag variant against baseline. +3. Take the top-3 single flags by RTF improvement with zero WER drift. +4. Combine pairwise. +5. Top-scored composite config wins. + +This gives us ~9–15 subprocess runs instead of the ~32 a full combinatorial sweep would need; converges on local optima without the combinatorial explosion. + +### Metrics + +- **Real-time factor (RTF)** = `audio_seconds / inference_wall_seconds`. Lower is better. +- **Word error rate (WER)** against the ground-truth transcript. Any config with >0.5% WER drift from baseline is rejected regardless of RTF improvement. +- **Peak VRAM** (optional, best-effort via `nvidia-smi` / `rocm-smi` sampling). + +### Runtime + +~5–15 minutes on typical hardware. Progress bar + ETA rendered to stderr so stdout stays machine-readable. + +### Bundled fixture + +A 20-second public-domain speech clip with a known-good reference transcript, committed to `crates/bench/fixtures/`. Source: LibriVox recording (CC0). + +### Output schema (`gpu-profile.toml`) + +```toml +[benchmarked_at] +timestamp = "2026-04-21T14:32:00Z" +kon_version = "0.1.0" +model = "whisper-distil-large-v3" + +[hardware] +gpu_name = "NVIDIA GeForce RTX 4070" +vram_mb = 12282 +driver = "nvidia 550.120" +os = "linux" +mesa = "" + +[baseline] +rtf = 0.043 +wer = 0.028 + +[best] +rtf = 0.031 +rtf_improvement = 0.279 # 27.9% faster +wer = 0.028 + +[best.env] +GGML_VK_DISABLE_COOPMAT = "0" +GGML_VK_FORCE_FP32 = "0" +# … full flag set, including unchanged ones, for reproducibility +``` + +### Crate layout + +``` +crates/bench/ +├── Cargo.toml +├── fixtures/ +│ ├── librivox-sample.wav +│ └── librivox-sample.txt +└── src/ + ├── main.rs # CLI + parent process + ├── runner.rs # subprocess harness (child entry gate: KON_BENCH_RUN=1) + ├── matrix.rs # grid-search + top-k logic + ├── metrics.rs # RTF + WER + optional VRAM sampling + └── profile.rs # TOML serialise +``` + +Depends on `kon-transcription` + `kon-llm` + `kon-audio` as path deps so it reuses the existing model-loading code. + +### Acceptance + +- `kon-bench --quick` runs unattended to completion on a fresh install +- Produces a valid `gpu-profile.toml` +- "Apply auto-tuned" button in Settings consumes the TOML and updates Phase 1 toggles (restart banner fires as expected) +- Re-running with `--compare ` produces reproducible-enough numbers (RTF within 5% run-to-run) + +--- + +## Phase 3-lite — `kon-configs` community repo + +**Effort**: 3 days (1 for repo + seeds, 2 for Kon-side fetch + apply UI). + +**What ships**: A separate public GitHub repo `kon-configs` (not part of the kon main repo) seeded with 2–3 curated configs. Kon's Settings page gets a "Browse community configs" button that fetches matching configs for the user's detected hardware. + +### Repo structure + +``` +kon-configs/ +├── README.md # pitch + how to benefit +├── CONTRIBUTING.md # required fields, benchmark protocol, fork/PR flow +├── SCHEMA.md # TOML schema documentation +├── index.json # manifest for Kon to discover configs +└── configs/ + ├── nvidia/ + │ ├── rtx-3060-12gb-linux.toml + │ └── rtx-4070-linux.toml + ├── amd/ + │ └── rx-6700xt-mesa-23-linux.toml + └── intel/ + └── arc-a770-windows.toml +``` + +### Config TOML + +Extends Phase 2's `gpu-profile.toml` schema with an `[attribution]` section: + +```toml +[attribution] +submitter = "@username" +notes = "Tested with 1-hour continuous dictation session, no crashes." +``` + +### Contribution flow (manual, honour-system MVP) + +1. User runs `kon-bench` on their hardware. +2. User runs `kon-bench --compare` against baseline to confirm improvement isn't noise. +3. User forks `kon-configs`, commits their TOML under `configs//`, opens PR. +4. Maintainer reviews format + plausibility, merges. +5. No CI replay — revisit if spam becomes a problem. + +### Kon integration + +- New Tauri command `fetch_community_configs(gpu_fingerprint)` — HTTPS GET `https://raw.githubusercontent.com//kon-configs/main/index.json` for the manifest, then fetches matching TOMLs +- Fingerprint match: GPU name substring + VRAM tier (e.g., `"RTX 3060"` + `"12gb"`) +- Settings "Browse community configs" button lists matches with submitter, claimed RTF improvement, and a preview of the toggle deltas +- Applying a config updates Phase 1 toggles AND stores provenance (source = `"community"`, submitter, fetch date) + +### What we explicitly skip at MVP + +- **No CI replay**. Maintainer eyeballs + honour system. Revisit past ~50 configs or on abuse. +- **No automated upload from `kon-bench`**. User always commits + PRs manually. Zero privacy concerns, zero spam surface. +- **No sophisticated fingerprint normalisation**. Substring matching is sufficient. + +### Acceptance + +- Repo exists with README + CONTRIBUTING + 2–3 seed configs +- Kon Settings fetches + lists + applies a community config end-to-end +- "Revert to default" path works (Phase 1's reset) + +--- + +## User experience — the one-click path + +This is the UX the three phases together enable. All three are prerequisites; Phase 3-lite is what turns "run a CLI" into "click a button." + +### First-launch onboarding nudge + +After the existing first-run model download, Kon surfaces a non-modal card: + +``` +🎛 GPU Optimisation +Detected: NVIDIA RTX 4070 (12 GB) · Linux Wayland +Current: Default GGML kernels + +[ Auto-optimise ] [ Show advanced ] [ Skip ] +``` + +"Auto-optimise" triggers the hybrid flow below. "Show advanced" expands the Phase 1 toggle panel directly. "Skip" dismisses; user can always come back via Settings. + +### The "Auto-optimise" flow + +Two steps, in this order: + +**Step 1 — Community config check (instant, ~2 s)** + +Kon fingerprints the GPU and queries the `kon-configs` manifest for matches. If a match exists, a preview card appears: + +``` +┌─────────────────────────────────────────────┐ +│ Community config available │ +│ │ +│ From: @someuser │ +│ Claimed: 27% faster · 0% accuracy drift │ +│ Tested: 2026-04-21, driver nvidia 550 │ +│ │ +│ Changes 2 settings: │ +│ • Cooperative matrix: on → off │ +│ • Integer dot product: on → off │ +│ │ +│ [ Apply (restart required) ] [ Cancel ] │ +└─────────────────────────────────────────────┘ +``` + +Apply → settings persist → restart prompt → done. 15 seconds end-to-end. + +**Step 2 — Fallback to local benchmark** + +If no community match, or the user prefers their own measurement: + +``` +┌─────────────────────────────────────────────┐ +│ No community config for your hardware yet │ +│ │ +│ We can benchmark your machine to find the │ +│ best settings. Takes ~8 minutes; runs in │ +│ the background while you keep using Kon. │ +│ │ +│ [ Benchmark my GPU ] [ Skip ] │ +└─────────────────────────────────────────────┘ +``` + +Kicks off `kon-bench` as a background process. Kon keeps working during the run. + +### Progress UI during benchmark + +Non-modal. Status chip in the lower-right of the main window: + +``` +⚙ Benchmarking GPU · 4 of 12 tested · ~5 min remaining [ cancel ] +``` + +On completion, a toast: + +``` +Your GPU is 27% faster with the new config. [ Review → ] +``` + +Review opens the same preview card as the community-config flow, with the same Apply / Cancel options. + +### After applying + +Settings shows the active config's provenance: + +- `Using community config · applied 2026-04-21 · by @someuser` +- `Using auto-tuned config · benchmarked 2026-04-21` +- `Using defaults` + +Plus a "Revert to previous config" button, active for 7 days after any change, in case the new config misbehaves in real use (silent accuracy drift, crashes on long sessions, etc.) that the benchmark didn't catch. + +### Optional — sharing back to the community + +After a successful local benchmark that shows meaningful gains, Kon prompts: + +``` +┌─────────────────────────────────────────────┐ +│ Share your config with the community? │ +│ │ +│ Your RTX 4070 tuning got you 27% faster. │ +│ Other RTX 4070 users would benefit. │ +│ │ +│ Shared data: GPU name, driver version, OS, │ +│ config flags, benchmark numbers. │ +│ NOT shared: personal info, audio, anything │ +│ that identifies you beyond the GitHub fork. │ +│ │ +│ [ Review payload ] [ Create PR ] [ No ] │ +└─────────────────────────────────────────────┘ +``` + +"Create PR" opens the user's browser to `github.com/…/kon-configs/new/main` with the TOML prefilled in the PR body. User finishes the submission on GitHub (still honour-system; no automated uploads, no telemetry). + +### Non-GPU / integrated-only fallback + +If `sysinfo` reports no dedicated GPU or Vulkan isn't available, the card replaces itself with: + +``` +🎛 GPU Optimisation +No dedicated GPU detected — Kon is using CPU inference. +GPU tuning doesn't apply to this setup. +``` + +No nag, no hidden settings, no broken experience. + +### Yes, "one click" is achievable + +For users whose GPU has a community-contributed config, the experience is **literally one click** (the Apply button), plus a restart. ~15 seconds. + +For users without a community match, the experience is **two clicks** (trigger bench → apply results on completion), with a passive ~8-minute background wait in between. + +For users on integrated graphics / no GPU, the experience is **zero clicks** — Kon tells them GPU tuning doesn't apply and moves on. + +--- + +## Sequencing + +Strict linear: Phase 1 → Phase 2 → Phase 3-lite. Each phase merges to `main` and gets dogfooded before the next starts. + +- Phase 1 is a prereq for Phase 2 — `kon-bench`'s output needs the Phase 1 settings schema to be its consumption target. +- Phase 2 is a prereq for Phase 3-lite — the community repo's config TOML schema **is** Phase 2's output schema (with an added `[attribution]` section). + +## Shelved with rationale + +- **Phase 4 — custom SPIR-V shader drops.** Blocked on `ggml`-dedup workstream. Pinned in memory. +- **Phase 5 — agentic (Karpathy-style) autotune.** Phase 2's grid search produces schema-compatible results, so Phase 5 can drop in later without a schema break. Pinned. +- **Phase 3's CI replay.** Defer until spam / bad-config abuse is a real problem rather than a hypothetical one. Honour-system PR review is sufficient for the MVP community. +- **`kon-bench` automated upload.** Deliberately manual for MVP — removes all privacy / spam / rate-limiting concerns. Revisit when the community volume justifies the infrastructure.