# Kon — GPU Tuning & Community Config Plan *Implementation spec for the first three phases of the GPU kernel tuning roadmap. The full five-phase roadmap is pinned in memory; this document scopes the MVP subset that ships real value without pulling in `ggml`-dedup or agentic-search prerequisites.* ## Scope **IN** (this document): - Phase 1 — Advanced GPU tuning settings panel (exposing GGML env vars) - Phase 2 — `kon-bench` local autotuning CLI - Phase 3-lite — `kon-configs` community repo with manual-PR workflow (no CI replay) **OUT** (pinned to memory for later): - Phase 4 — custom SPIR-V shader drops (blocked on `ggml`-dedup) - Phase 5 — Karpathy-style agentic autotuning - CI replay for community repo (defer until spam / bad configs become a real problem) This subset captures roughly 85% of the perceived value for ~20% of the total effort. The deferred pieces are where complexity explodes; the MVP stops before it. --- ## Phase 1 — Advanced GPU tuning settings panel **Effort**: 1–2 days. **What ships**: Settings → Advanced → GPU Tuning collapsible section with toggles for GGML env vars. Env vars are applied at app startup before any GPU backend initialises. Per-profile storage; restart required to take effect. ### Toggles shipped at MVP | UI label | Env var | Default | When users enable | |---|---|---|---| | Disable cooperative matrix | `GGML_VK_DISABLE_COOPMAT` | off | "Inference hangs" on RDNA2 / buggy Mesa versions | | Force FP32 math | `GGML_VK_FORCE_FP32` | off | "Garbage transcripts" on Intel Arc / older NVIDIA | | Disable FP16 ops | `GGML_VK_DISABLE_F16` | off | Silent-fail on some Mesa 22.x builds | | Disable integer dot product | `GGML_VK_DISABLE_INTEGER_DOT_PRODUCT` | off | "Random NaN" on RDNA2 with certain drivers | | Enable Vulkan validation | `GGML_VK_VALIDATE` | off | Diagnostic only; impacts performance | Metal / CUDA counterparts slot in when those backends grow in Kon. Today Kon is Vulkan-only. ### Design - New `SettingsState.gpuTuning: { disableCoopmat: boolean, forceFp32: boolean, disableF16: boolean, disableIntegerDotProduct: boolean, enableValidation: boolean }` in [src/lib/types/app.ts](../../src/lib/types/app.ts) - All defaults `false` in [src/lib/stores/page.svelte.ts](../../src/lib/stores/page.svelte.ts) - Persistence uses the existing `save_preferences` → SQLite `kon_preferences` path - Backend reads preferences at the **very top** of `run()` in [src-tauri/src/lib.rs](../../src-tauri/src/lib.rs) — before `tauri::Builder::default()` spawns threads — and writes via `unsafe { std::env::set_var(...) }`. Matches the existing `ensure_x11_on_wayland` pattern - Settings UI shows a sticky "Restart required for changes to take effect" banner when any toggle has drifted from its launch-time value - A "Reset to defaults" button zeroes all toggles ### Acceptance - Toggling "Disable cooperative matrix" on and restarting → `vulkaninfo` (or GGML debug logs) confirms the knob is honoured at backend init - Default all-off produces identical performance + transcription output to the current `main` (smoke test) - An integration test with a fake settings fixture confirms env vars are set before `AppState` initialises --- ## Phase 2 — `kon-bench` local autotuning CLI **Effort**: 3–5 days. **What ships**: New workspace binary `crates/bench/` producing a `kon-bench` executable. User runs it once post-install; output lands at `~/.kon/gpu-profile.toml` with the best-scoring config for their hardware. Settings page gets an "Apply auto-tuned profile" button that consumes the TOML and updates the Phase 1 toggles. ### CLI surface ``` kon-bench --quick # bundled 20s sample + reference transcript kon-bench --model --audio --transcript kon-bench --compare # benchmark a specific profile vs default ``` ### Execution model Grid-search via **subprocess spawning**. Each config variant runs in a child process with its own env vars — because env vars must be set at process startup; you cannot safely mutate GGML's runtime state once it's initialised. The parent serialises variants, spawns a child per variant, waits for each to exit with a JSON line on stdout, aggregates and ranks. ### Search strategy (not naive combinatorial) 1. Run baseline (all defaults). 2. Run each single-flag variant against baseline. 3. Take the top-3 single flags by RTF improvement with zero WER drift. 4. Combine pairwise. 5. Top-scored composite config wins. This gives us ~9–15 subprocess runs instead of the ~32 a full combinatorial sweep would need; converges on local optima without the combinatorial explosion. ### Metrics - **Real-time factor (RTF)** = `audio_seconds / inference_wall_seconds`. Lower is better. - **Word error rate (WER)** against the ground-truth transcript. Any config with >0.5% WER drift from baseline is rejected regardless of RTF improvement. - **Peak VRAM** (optional, best-effort via `nvidia-smi` / `rocm-smi` sampling). ### Runtime ~5–15 minutes on typical hardware. Progress bar + ETA rendered to stderr so stdout stays machine-readable. ### Bundled fixture A 20-second public-domain speech clip with a known-good reference transcript, committed to `crates/bench/fixtures/`. Source: LibriVox recording (CC0). ### Output schema (`gpu-profile.toml`) ```toml [benchmarked_at] timestamp = "2026-04-21T14:32:00Z" kon_version = "0.1.0" model = "whisper-distil-large-v3" [hardware] gpu_name = "NVIDIA GeForce RTX 4070" vram_mb = 12282 driver = "nvidia 550.120" os = "linux" mesa = "" [baseline] rtf = 0.043 wer = 0.028 [best] rtf = 0.031 rtf_improvement = 0.279 # 27.9% faster wer = 0.028 [best.env] GGML_VK_DISABLE_COOPMAT = "0" GGML_VK_FORCE_FP32 = "0" # … full flag set, including unchanged ones, for reproducibility ``` ### Crate layout ``` crates/bench/ ├── Cargo.toml ├── fixtures/ │ ├── librivox-sample.wav │ └── librivox-sample.txt └── src/ ├── main.rs # CLI + parent process ├── runner.rs # subprocess harness (child entry gate: KON_BENCH_RUN=1) ├── matrix.rs # grid-search + top-k logic ├── metrics.rs # RTF + WER + optional VRAM sampling └── profile.rs # TOML serialise ``` Depends on `kon-transcription` + `kon-llm` + `kon-audio` as path deps so it reuses the existing model-loading code. ### Acceptance - `kon-bench --quick` runs unattended to completion on a fresh install - Produces a valid `gpu-profile.toml` - "Apply auto-tuned" button in Settings consumes the TOML and updates Phase 1 toggles (restart banner fires as expected) - Re-running with `--compare ` produces reproducible-enough numbers (RTF within 5% run-to-run) --- ## Phase 3-lite — `kon-configs` community repo **Effort**: 3 days (1 for repo + seeds, 2 for Kon-side fetch + apply UI). **What ships**: A separate public GitHub repo `kon-configs` (not part of the kon main repo) seeded with 2–3 curated configs. Kon's Settings page gets a "Browse community configs" button that fetches matching configs for the user's detected hardware. ### Repo structure ``` kon-configs/ ├── README.md # pitch + how to benefit ├── CONTRIBUTING.md # required fields, benchmark protocol, fork/PR flow ├── SCHEMA.md # TOML schema documentation ├── index.json # manifest for Kon to discover configs └── configs/ ├── nvidia/ │ ├── rtx-3060-12gb-linux.toml │ └── rtx-4070-linux.toml ├── amd/ │ └── rx-6700xt-mesa-23-linux.toml └── intel/ └── arc-a770-windows.toml ``` ### Config TOML Extends Phase 2's `gpu-profile.toml` schema with an `[attribution]` section: ```toml [attribution] submitter = "@username" notes = "Tested with 1-hour continuous dictation session, no crashes." ``` ### Contribution flow (manual, honour-system MVP) 1. User runs `kon-bench` on their hardware. 2. User runs `kon-bench --compare` against baseline to confirm improvement isn't noise. 3. User forks `kon-configs`, commits their TOML under `configs//`, opens PR. 4. Maintainer reviews format + plausibility, merges. 5. No CI replay — revisit if spam becomes a problem. ### Kon integration - New Tauri command `fetch_community_configs(gpu_fingerprint)` — HTTPS GET `https://raw.githubusercontent.com//kon-configs/main/index.json` for the manifest, then fetches matching TOMLs - Fingerprint match: GPU name substring + VRAM tier (e.g., `"RTX 3060"` + `"12gb"`) - Settings "Browse community configs" button lists matches with submitter, claimed RTF improvement, and a preview of the toggle deltas - Applying a config updates Phase 1 toggles AND stores provenance (source = `"community"`, submitter, fetch date) ### What we explicitly skip at MVP - **No CI replay**. Maintainer eyeballs + honour system. Revisit past ~50 configs or on abuse. - **No automated upload from `kon-bench`**. User always commits + PRs manually. Zero privacy concerns, zero spam surface. - **No sophisticated fingerprint normalisation**. Substring matching is sufficient. ### Acceptance - Repo exists with README + CONTRIBUTING + 2–3 seed configs - Kon Settings fetches + lists + applies a community config end-to-end - "Revert to default" path works (Phase 1's reset) --- ## User experience — the one-click path This is the UX the three phases together enable. All three are prerequisites; Phase 3-lite is what turns "run a CLI" into "click a button." ### First-launch onboarding nudge After the existing first-run model download, Kon surfaces a non-modal card: ``` 🎛 GPU Optimisation Detected: NVIDIA RTX 4070 (12 GB) · Linux Wayland Current: Default GGML kernels [ Auto-optimise ] [ Show advanced ] [ Skip ] ``` "Auto-optimise" triggers the hybrid flow below. "Show advanced" expands the Phase 1 toggle panel directly. "Skip" dismisses; user can always come back via Settings. ### The "Auto-optimise" flow Two steps, in this order: **Step 1 — Community config check (instant, ~2 s)** Kon fingerprints the GPU and queries the `kon-configs` manifest for matches. If a match exists, a preview card appears: ``` ┌─────────────────────────────────────────────┐ │ Community config available │ │ │ │ From: @someuser │ │ Claimed: 27% faster · 0% accuracy drift │ │ Tested: 2026-04-21, driver nvidia 550 │ │ │ │ Changes 2 settings: │ │ • Cooperative matrix: on → off │ │ • Integer dot product: on → off │ │ │ │ [ Apply (restart required) ] [ Cancel ] │ └─────────────────────────────────────────────┘ ``` Apply → settings persist → restart prompt → done. 15 seconds end-to-end. **Step 2 — Fallback to local benchmark** If no community match, or the user prefers their own measurement: ``` ┌─────────────────────────────────────────────┐ │ No community config for your hardware yet │ │ │ │ We can benchmark your machine to find the │ │ best settings. Takes ~8 minutes; runs in │ │ the background while you keep using Kon. │ │ │ │ [ Benchmark my GPU ] [ Skip ] │ └─────────────────────────────────────────────┘ ``` Kicks off `kon-bench` as a background process. Kon keeps working during the run. ### Progress UI during benchmark Non-modal. Status chip in the lower-right of the main window: ``` ⚙ Benchmarking GPU · 4 of 12 tested · ~5 min remaining [ cancel ] ``` On completion, a toast: ``` Your GPU is 27% faster with the new config. [ Review → ] ``` Review opens the same preview card as the community-config flow, with the same Apply / Cancel options. ### After applying Settings shows the active config's provenance: - `Using community config · applied 2026-04-21 · by @someuser` - `Using auto-tuned config · benchmarked 2026-04-21` - `Using defaults` Plus a "Revert to previous config" button, active for 7 days after any change, in case the new config misbehaves in real use (silent accuracy drift, crashes on long sessions, etc.) that the benchmark didn't catch. ### Optional — sharing back to the community After a successful local benchmark that shows meaningful gains, Kon prompts: ``` ┌─────────────────────────────────────────────┐ │ Share your config with the community? │ │ │ │ Your RTX 4070 tuning got you 27% faster. │ │ Other RTX 4070 users would benefit. │ │ │ │ Shared data: GPU name, driver version, OS, │ │ config flags, benchmark numbers. │ │ NOT shared: personal info, audio, anything │ │ that identifies you beyond the GitHub fork. │ │ │ │ [ Review payload ] [ Create PR ] [ No ] │ └─────────────────────────────────────────────┘ ``` "Create PR" opens the user's browser to `github.com/…/kon-configs/new/main` with the TOML prefilled in the PR body. User finishes the submission on GitHub (still honour-system; no automated uploads, no telemetry). ### Non-GPU / integrated-only fallback If `sysinfo` reports no dedicated GPU or Vulkan isn't available, the card replaces itself with: ``` 🎛 GPU Optimisation No dedicated GPU detected — Kon is using CPU inference. GPU tuning doesn't apply to this setup. ``` No nag, no hidden settings, no broken experience. ### Yes, "one click" is achievable For users whose GPU has a community-contributed config, the experience is **literally one click** (the Apply button), plus a restart. ~15 seconds. For users without a community match, the experience is **two clicks** (trigger bench → apply results on completion), with a passive ~8-minute background wait in between. For users on integrated graphics / no GPU, the experience is **zero clicks** — Kon tells them GPU tuning doesn't apply and moves on. --- ## Sequencing Strict linear: Phase 1 → Phase 2 → Phase 3-lite. Each phase merges to `main` and gets dogfooded before the next starts. - Phase 1 is a prereq for Phase 2 — `kon-bench`'s output needs the Phase 1 settings schema to be its consumption target. - Phase 2 is a prereq for Phase 3-lite — the community repo's config TOML schema **is** Phase 2's output schema (with an added `[attribution]` section). ## Shelved with rationale - **Phase 4 — custom SPIR-V shader drops.** Blocked on `ggml`-dedup workstream. Pinned in memory. - **Phase 5 — agentic (Karpathy-style) autotune.** Phase 2's grid search produces schema-compatible results, so Phase 5 can drop in later without a schema break. Pinned. - **Phase 3's CI replay.** Defer until spam / bad-config abuse is a real problem rather than a hypothetical one. Honour-system PR review is sufficient for the MVP community. - **`kon-bench` automated upload.** Deliberately manual for MVP — removes all privacy / spam / rate-limiting concerns. Revisit when the community volume justifies the infrastructure.