jake/Lumotia
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(gpu-tuning): add MVP plan — three phases with one-click UX
Some checks failed
check / cargo check (macos-latest) (push) Has been cancelled
Details
check / cargo check (ubuntu-22.04) (push) Has been cancelled
Details
check / cargo check (windows-latest) (push) Has been cancelled
Details
check / svelte build + lint (push) Has been cancelled
Details

Browse Source 
Specs the subset of the five-phase GPU kernel tuning roadmap that ships
without requiring ggml-dedup or agentic-search prerequisites:

- Phase 1 — Advanced → GPU Tuning settings panel (GGML env var toggles,
  applied at startup before threads spawn).
- Phase 2 — kon-bench local autotuning CLI. Subprocess-based grid search
  over env vars, outputs a ranked gpu-profile.toml.
- Phase 3-lite — kon-configs community repo. Manual-PR workflow (no CI
  replay), fingerprint-matched fetch from Kon Settings.

Total ~7–10 days of focused work; captures roughly 85% of the eventual
value of the full roadmap. Phases 4–5 (custom SPIR-V drops + agentic
autotune) stay pinned in memory.

Includes the UX spec for the "one-click auto-optimise" flow: community
config check first (~15 s end-to-end), local benchmark fallback
(~8 min backgrounded), opt-in share-back via browser PR. Non-GPU users
see a clean "tuning doesn't apply" card with no nag.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jake
2026-04-21 12:32:43 +01:00
parent ff22497468
commit 65280c776e
1 changed files with 364 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

364
docs/gpu-tuning/plan.md Normal file
View File

@@ -0,0 +1,364 @@
# Kon — GPU Tuning & Community Config Plan
*Implementation spec for the first three phases of the GPU kernel tuning roadmap. The full five-phase roadmap is pinned in memory; this document scopes the MVP subset that ships real value without pulling in `ggml`-dedup or agentic-search prerequisites.*
## Scope
**IN** (this document):
- Phase 1 — Advanced GPU tuning settings panel (exposing GGML env vars)
- Phase 2 — `kon-bench` local autotuning CLI
- Phase 3-lite — `kon-configs` community repo with manual-PR workflow (no CI replay)
**OUT** (pinned to memory for later):
- Phase 4 — custom SPIR-V shader drops (blocked on `ggml`-dedup)
- Phase 5 — Karpathy-style agentic autotuning
- CI replay for community repo (defer until spam / bad configs become a real problem)
This subset captures roughly 85% of the perceived value for ~20% of the total effort. The deferred pieces are where complexity explodes; the MVP stops before it.
---
## Phase 1 — Advanced GPU tuning settings panel
**Effort**: 1–2 days.
**What ships**: Settings → Advanced → GPU Tuning collapsible section with toggles for GGML env vars. Env vars are applied at app startup before any GPU backend initialises. Per-profile storage; restart required to take effect.
### Toggles shipped at MVP
| UI label | Env var | Default | When users enable |
|---|---|---|---|
| Disable cooperative matrix | `GGML_VK_DISABLE_COOPMAT` | off | "Inference hangs" on RDNA2 / buggy Mesa versions |
| Force FP32 math | `GGML_VK_FORCE_FP32` | off | "Garbage transcripts" on Intel Arc / older NVIDIA |
| Disable FP16 ops | `GGML_VK_DISABLE_F16` | off | Silent-fail on some Mesa 22.x builds |
| Disable integer dot product | `GGML_VK_DISABLE_INTEGER_DOT_PRODUCT` | off | "Random NaN" on RDNA2 with certain drivers |
| Enable Vulkan validation | `GGML_VK_VALIDATE` | off | Diagnostic only; impacts performance |
Metal / CUDA counterparts slot in when those backends grow in Kon. Today Kon is Vulkan-only.
### Design
- New `SettingsState.gpuTuning: { disableCoopmat: boolean, forceFp32: boolean, disableF16: boolean, disableIntegerDotProduct: boolean, enableValidation: boolean }` in [src/lib/types/app.ts](../../src/lib/types/app.ts)
- All defaults `false` in [src/lib/stores/page.svelte.ts](../../src/lib/stores/page.svelte.ts)
- Persistence uses the existing `save_preferences` → SQLite `kon_preferences` path
- Backend reads preferences at the **very top** of `run()` in [src-tauri/src/lib.rs](../../src-tauri/src/lib.rs) — before `tauri::Builder::default()` spawns threads — and writes via `unsafe { std::env::set_var(...) }`. Matches the existing `ensure_x11_on_wayland` pattern
- Settings UI shows a sticky "Restart required for changes to take effect" banner when any toggle has drifted from its launch-time value
- A "Reset to defaults" button zeroes all toggles
### Acceptance
- Toggling "Disable cooperative matrix" on and restarting → `vulkaninfo` (or GGML debug logs) confirms the knob is honoured at backend init
- Default all-off produces identical performance + transcription output to the current `main` (smoke test)
- An integration test with a fake settings fixture confirms env vars are set before `AppState` initialises
---
## Phase 2 — `kon-bench` local autotuning CLI
**Effort**: 3–5 days.
**What ships**: New workspace binary `crates/bench/` producing a `kon-bench` executable. User runs it once post-install; output lands at `~/.kon/gpu-profile.toml` with the best-scoring config for their hardware. Settings page gets an "Apply auto-tuned profile" button that consumes the TOML and updates the Phase 1 toggles.
### CLI surface
```
kon-bench --quick # bundled 20s sample + reference transcript
kon-bench --model <path> --audio <wav> --transcript <txt>
kon-bench --compare <profile.toml> # benchmark a specific profile vs default
```
### Execution model
Grid-search via **subprocess spawning**. Each config variant runs in a child process with its own env vars — because env vars must be set at process startup; you cannot safely mutate GGML's runtime state once it's initialised. The parent serialises variants, spawns a child per variant, waits for each to exit with a JSON line on stdout, aggregates and ranks.
### Search strategy (not naive combinatorial)
1. Run baseline (all defaults).
2. Run each single-flag variant against baseline.
3. Take the top-3 single flags by RTF improvement with zero WER drift.
4. Combine pairwise.
5. Top-scored composite config wins.
This gives us ~9–15 subprocess runs instead of the ~32 a full combinatorial sweep would need; converges on local optima without the combinatorial explosion.
### Metrics
- **Real-time factor (RTF)** = `audio_seconds / inference_wall_seconds`. Lower is better.
- **Word error rate (WER)** against the ground-truth transcript. Any config with >0.5% WER drift from baseline is rejected regardless of RTF improvement.
- **Peak VRAM** (optional, best-effort via `nvidia-smi` / `rocm-smi` sampling).
### Runtime
~5–15 minutes on typical hardware. Progress bar + ETA rendered to stderr so stdout stays machine-readable.
### Bundled fixture
A 20-second public-domain speech clip with a known-good reference transcript, committed to `crates/bench/fixtures/`. Source: LibriVox recording (CC0).
### Output schema (`gpu-profile.toml`)
```toml
[benchmarked_at]
timestamp = "2026-04-21T14:32:00Z"
kon_version = "0.1.0"
model = "whisper-distil-large-v3"
[hardware]
gpu_name = "NVIDIA GeForce RTX 4070"
vram_mb = 12282
driver = "nvidia 550.120"
os = "linux"
mesa = ""
[baseline]
rtf = 0.043
wer = 0.028
[best]
rtf = 0.031
rtf_improvement = 0.279 # 27.9% faster
wer = 0.028
[best.env]
GGML_VK_DISABLE_COOPMAT = "0"
GGML_VK_FORCE_FP32 = "0"
# … full flag set, including unchanged ones, for reproducibility
```
### Crate layout
```
crates/bench/
├── Cargo.toml
├── fixtures/
│ ├── librivox-sample.wav
│ └── librivox-sample.txt
└── src/
├── main.rs # CLI + parent process
├── runner.rs # subprocess harness (child entry gate: KON_BENCH_RUN=1)
├── matrix.rs # grid-search + top-k logic
├── metrics.rs # RTF + WER + optional VRAM sampling
└── profile.rs # TOML serialise
```
Depends on `kon-transcription` + `kon-llm` + `kon-audio` as path deps so it reuses the existing model-loading code.
### Acceptance
- `kon-bench --quick` runs unattended to completion on a fresh install
- Produces a valid `gpu-profile.toml`
- "Apply auto-tuned" button in Settings consumes the TOML and updates Phase 1 toggles (restart banner fires as expected)
- Re-running with `--compare <profile>` produces reproducible-enough numbers (RTF within 5% run-to-run)
---
## Phase 3-lite — `kon-configs` community repo
**Effort**: 3 days (1 for repo + seeds, 2 for Kon-side fetch + apply UI).
**What ships**: A separate public GitHub repo `kon-configs` (not part of the kon main repo) seeded with 2–3 curated configs. Kon's Settings page gets a "Browse community configs" button that fetches matching configs for the user's detected hardware.
### Repo structure
```
kon-configs/
├── README.md # pitch + how to benefit
├── CONTRIBUTING.md # required fields, benchmark protocol, fork/PR flow
├── SCHEMA.md # TOML schema documentation
├── index.json # manifest for Kon to discover configs
└── configs/
├── nvidia/
│ ├── rtx-3060-12gb-linux.toml
│ └── rtx-4070-linux.toml
├── amd/
│ └── rx-6700xt-mesa-23-linux.toml
└── intel/
└── arc-a770-windows.toml
```
### Config TOML
Extends Phase 2's `gpu-profile.toml` schema with an `[attribution]` section:
```toml
[attribution]
submitter = "@username"
notes = "Tested with 1-hour continuous dictation session, no crashes."
```
### Contribution flow (manual, honour-system MVP)
1. User runs `kon-bench` on their hardware.
2. User runs `kon-bench --compare` against baseline to confirm improvement isn't noise.
3. User forks `kon-configs`, commits their TOML under `configs/<vendor>/`, opens PR.
4. Maintainer reviews format + plausibility, merges.
5. No CI replay — revisit if spam becomes a problem.
### Kon integration
- New Tauri command `fetch_community_configs(gpu_fingerprint)` — HTTPS GET `https://raw.githubusercontent.com/<org>/kon-configs/main/index.json` for the manifest, then fetches matching TOMLs
- Fingerprint match: GPU name substring + VRAM tier (e.g., `"RTX 3060"` + `"12gb"`)
- Settings "Browse community configs" button lists matches with submitter, claimed RTF improvement, and a preview of the toggle deltas
- Applying a config updates Phase 1 toggles AND stores provenance (source = `"community"`, submitter, fetch date)
### What we explicitly skip at MVP
- **No CI replay**. Maintainer eyeballs + honour system. Revisit past ~50 configs or on abuse.
- **No automated upload from `kon-bench`**. User always commits + PRs manually. Zero privacy concerns, zero spam surface.
- **No sophisticated fingerprint normalisation**. Substring matching is sufficient.
### Acceptance
- Repo exists with README + CONTRIBUTING + 2–3 seed configs
- Kon Settings fetches + lists + applies a community config end-to-end
- "Revert to default" path works (Phase 1's reset)
---
## User experience — the one-click path
This is the UX the three phases together enable. All three are prerequisites; Phase 3-lite is what turns "run a CLI" into "click a button."
### First-launch onboarding nudge
After the existing first-run model download, Kon surfaces a non-modal card:
```
🎛 GPU Optimisation
Detected: NVIDIA RTX 4070 (12 GB) · Linux Wayland
Current: Default GGML kernels
[ Auto-optimise ] [ Show advanced ] [ Skip ]
```
"Auto-optimise" triggers the hybrid flow below. "Show advanced" expands the Phase 1 toggle panel directly. "Skip" dismisses; user can always come back via Settings.
### The "Auto-optimise" flow
Two steps, in this order:
**Step 1 — Community config check (instant, ~2 s)**
Kon fingerprints the GPU and queries the `kon-configs` manifest for matches. If a match exists, a preview card appears:
```
┌─────────────────────────────────────────────┐
│ Community config available │
│ │
│ From: @someuser │
│ Claimed: 27% faster · 0% accuracy drift │
│ Tested: 2026-04-21, driver nvidia 550 │
│ │
│ Changes 2 settings: │
│ • Cooperative matrix: on → off │
│ • Integer dot product: on → off │
│ │
│ [ Apply (restart required) ] [ Cancel ] │
└─────────────────────────────────────────────┘
```
Apply → settings persist → restart prompt → done. 15 seconds end-to-end.
**Step 2 — Fallback to local benchmark**
If no community match, or the user prefers their own measurement:
```
┌─────────────────────────────────────────────┐
│ No community config for your hardware yet │
│ │
│ We can benchmark your machine to find the │
│ best settings. Takes ~8 minutes; runs in │
│ the background while you keep using Kon. │
│ │
│ [ Benchmark my GPU ] [ Skip ] │
└─────────────────────────────────────────────┘
```
Kicks off `kon-bench` as a background process. Kon keeps working during the run.
### Progress UI during benchmark
Non-modal. Status chip in the lower-right of the main window:
```
⚙ Benchmarking GPU · 4 of 12 tested · ~5 min remaining [ cancel ]
```
On completion, a toast:
```
Your GPU is 27% faster with the new config. [ Review → ]
```
Review opens the same preview card as the community-config flow, with the same Apply / Cancel options.
### After applying
Settings shows the active config's provenance:
- `Using community config · applied 2026-04-21 · by @someuser`
- `Using auto-tuned config · benchmarked 2026-04-21`
- `Using defaults`
Plus a "Revert to previous config" button, active for 7 days after any change, in case the new config misbehaves in real use (silent accuracy drift, crashes on long sessions, etc.) that the benchmark didn't catch.
### Optional — sharing back to the community
After a successful local benchmark that shows meaningful gains, Kon prompts:
```
┌─────────────────────────────────────────────┐
│ Share your config with the community? │
│ │
│ Your RTX 4070 tuning got you 27% faster. │
│ Other RTX 4070 users would benefit. │
│ │
│ Shared data: GPU name, driver version, OS, │
│ config flags, benchmark numbers. │
│ NOT shared: personal info, audio, anything │
│ that identifies you beyond the GitHub fork. │
│ │
│ [ Review payload ] [ Create PR ] [ No ] │
└─────────────────────────────────────────────┘
```
"Create PR" opens the user's browser to `github.com/…/kon-configs/new/main` with the TOML prefilled in the PR body. User finishes the submission on GitHub (still honour-system; no automated uploads, no telemetry).
### Non-GPU / integrated-only fallback
If `sysinfo` reports no dedicated GPU or Vulkan isn't available, the card replaces itself with:
```
🎛 GPU Optimisation
No dedicated GPU detected — Kon is using CPU inference.
GPU tuning doesn't apply to this setup.
```
No nag, no hidden settings, no broken experience.
### Yes, "one click" is achievable
For users whose GPU has a community-contributed config, the experience is **literally one click** (the Apply button), plus a restart. ~15 seconds.
For users without a community match, the experience is **two clicks** (trigger bench → apply results on completion), with a passive ~8-minute background wait in between.
For users on integrated graphics / no GPU, the experience is **zero clicks** — Kon tells them GPU tuning doesn't apply and moves on.
---
## Sequencing
Strict linear: Phase 1 → Phase 2 → Phase 3-lite. Each phase merges to `main` and gets dogfooded before the next starts.
- Phase 1 is a prereq for Phase 2 — `kon-bench`'s output needs the Phase 1 settings schema to be its consumption target.
- Phase 2 is a prereq for Phase 3-lite — the community repo's config TOML schema **is** Phase 2's output schema (with an added `[attribution]` section).
## Shelved with rationale
- **Phase 4 — custom SPIR-V shader drops.** Blocked on `ggml`-dedup workstream. Pinned in memory.
- **Phase 5 — agentic (Karpathy-style) autotune.** Phase 2's grid search produces schema-compatible results, so Phase 5 can drop in later without a schema break. Pinned.
- **Phase 3's CI replay.** Defer until spam / bad-config abuse is a real problem rather than a hypothetical one. Honour-system PR review is sufficient for the MVP community.
- **`kon-bench` automated upload.** Deliberately manual for MVP — removes all privacy / spam / rate-limiting concerns. Revisit when the community volume justifies the infrastructure.