Lumotia/docs/gpu-tuning/plan.md

Kon — GPU Tuning & Community Config Plan

Implementation spec for the first three phases of the GPU kernel tuning roadmap. The full five-phase roadmap is pinned in memory; this document scopes the MVP subset that ships real value without pulling in ggml-dedup or agentic-search prerequisites.

Scope

IN (this document):

• Phase 1 — Advanced GPU tuning settings panel (exposing GGML env vars)
• Phase 2 — kon-bench local autotuning CLI
• Phase 3-lite — kon-configs community repo with manual-PR workflow (no CI replay)

OUT (pinned to memory for later):

• Phase 4 — custom SPIR-V shader drops (blocked on ggml-dedup)
• Phase 5 — Karpathy-style agentic autotuning
• CI replay for community repo (defer until spam / bad configs become a real problem)

This subset captures roughly 85% of the perceived value for ~20% of the total effort. The deferred pieces are where complexity explodes; the MVP stops before it.

---

Phase 1 — Advanced GPU tuning settings panel

Effort: 1–2 days.

What ships: Settings → Advanced → GPU Tuning collapsible section with toggles for GGML env vars. Env vars are applied at app startup before any GPU backend initialises. Per-profile storage; restart required to take effect.

Toggles shipped at MVP

UI label	Env var	Default	When users enable
Disable cooperative matrix	GGML_VK_DISABLE_COOPMAT	off	"Inference hangs" on RDNA2 / buggy Mesa versions
Force FP32 math	GGML_VK_FORCE_FP32	off	"Garbage transcripts" on Intel Arc / older NVIDIA
Disable FP16 ops	GGML_VK_DISABLE_F16	off	Silent-fail on some Mesa 22.x builds
Disable integer dot product	GGML_VK_DISABLE_INTEGER_DOT_PRODUCT	off	"Random NaN" on RDNA2 with certain drivers
Enable Vulkan validation	GGML_VK_VALIDATE	off	Diagnostic only; impacts performance

Metal / CUDA counterparts slot in when those backends grow in Kon. Today Kon is Vulkan-only.

Design

• New SettingsState.gpuTuning: { disableCoopmat: boolean, forceFp32: boolean, disableF16: boolean, disableIntegerDotProduct: boolean, enableValidation: boolean } in src/lib/types/app.ts
• All defaults false in src/lib/stores/page.svelte.ts
• Persistence uses the existing save_preferences → SQLite kon_preferences path
• Backend reads preferences at the very top of run() in src-tauri/src/lib.rs — before tauri::Builder::default() spawns threads — and writes via unsafe { std::env::set_var(...) }. Matches the existing ensure_x11_on_wayland pattern
• Settings UI shows a sticky "Restart required for changes to take effect" banner when any toggle has drifted from its launch-time value
• A "Reset to defaults" button zeroes all toggles

Acceptance

• Toggling "Disable cooperative matrix" on and restarting → vulkaninfo (or GGML debug logs) confirms the knob is honoured at backend init
• Default all-off produces identical performance + transcription output to the current main (smoke test)
• An integration test with a fake settings fixture confirms env vars are set before AppState initialises

---

Phase 2 — kon-bench local autotuning CLI

Effort: 3–5 days.

What ships: New workspace binary crates/bench/ producing a kon-bench executable. User runs it once post-install; output lands at ~/.kon/gpu-profile.toml with the best-scoring config for their hardware. Settings page gets an "Apply auto-tuned profile" button that consumes the TOML and updates the Phase 1 toggles.

CLI surface

kon-bench --quick                      # bundled 20s sample + reference transcript
kon-bench --model <path> --audio <wav> --transcript <txt>
kon-bench --compare <profile.toml>     # benchmark a specific profile vs default

Execution model

Grid-search via subprocess spawning. Each config variant runs in a child process with its own env vars — because env vars must be set at process startup; you cannot safely mutate GGML's runtime state once it's initialised. The parent serialises variants, spawns a child per variant, waits for each to exit with a JSON line on stdout, aggregates and ranks.

Search strategy (not naive combinatorial)

1. Run baseline (all defaults).
2. Run each single-flag variant against baseline.
3. Take the top-3 single flags by RTF improvement with zero WER drift.
4. Combine pairwise.
5. Top-scored composite config wins.

This gives us ~9–15 subprocess runs instead of the ~32 a full combinatorial sweep would need; converges on local optima without the combinatorial explosion.

Metrics

• Real-time factor (RTF) = audio_seconds / inference_wall_seconds. Lower is better.
• Word error rate (WER) against the ground-truth transcript. Any config with >0.5% WER drift from baseline is rejected regardless of RTF improvement.
• Peak VRAM (optional, best-effort via nvidia-smi / rocm-smi sampling).

Runtime

~5–15 minutes on typical hardware. Progress bar + ETA rendered to stderr so stdout stays machine-readable.

Bundled fixture

A 20-second public-domain speech clip with a known-good reference transcript, committed to crates/bench/fixtures/. Source: LibriVox recording (CC0).

Output schema (gpu-profile.toml)

[benchmarked_at]
timestamp = "2026-04-21T14:32:00Z"
kon_version = "0.1.0"
model = "whisper-distil-large-v3"

[hardware]
gpu_name = "NVIDIA GeForce RTX 4070"
vram_mb = 12282
driver = "nvidia 550.120"
os = "linux"
mesa = ""

[baseline]
rtf = 0.043
wer = 0.028

[best]
rtf = 0.031
rtf_improvement = 0.279    # 27.9% faster
wer = 0.028

[best.env]
GGML_VK_DISABLE_COOPMAT = "0"
GGML_VK_FORCE_FP32 = "0"
# … full flag set, including unchanged ones, for reproducibility

Crate layout

crates/bench/
├── Cargo.toml
├── fixtures/
│   ├── librivox-sample.wav
│   └── librivox-sample.txt
└── src/
    ├── main.rs         # CLI + parent process
    ├── runner.rs       # subprocess harness (child entry gate: KON_BENCH_RUN=1)
    ├── matrix.rs       # grid-search + top-k logic
    ├── metrics.rs      # RTF + WER + optional VRAM sampling
    └── profile.rs      # TOML serialise

Depends on kon-transcription + kon-llm + kon-audio as path deps so it reuses the existing model-loading code.

Acceptance

• kon-bench --quick runs unattended to completion on a fresh install
• Produces a valid gpu-profile.toml
• "Apply auto-tuned" button in Settings consumes the TOML and updates Phase 1 toggles (restart banner fires as expected)
• Re-running with --compare <profile> produces reproducible-enough numbers (RTF within 5% run-to-run)

---

Phase 3-lite — kon-configs community repo

Effort: 3 days (1 for repo + seeds, 2 for Kon-side fetch + apply UI).

What ships: A separate public GitHub repo kon-configs (not part of the kon main repo) seeded with 2–3 curated configs. Kon's Settings page gets a "Browse community configs" button that fetches matching configs for the user's detected hardware.

Repo structure

kon-configs/
├── README.md              # pitch + how to benefit
├── CONTRIBUTING.md        # required fields, benchmark protocol, fork/PR flow
├── SCHEMA.md              # TOML schema documentation
├── index.json             # manifest for Kon to discover configs
└── configs/
    ├── nvidia/
    │   ├── rtx-3060-12gb-linux.toml
    │   └── rtx-4070-linux.toml
    ├── amd/
    │   └── rx-6700xt-mesa-23-linux.toml
    └── intel/
        └── arc-a770-windows.toml

Config TOML

Extends Phase 2's gpu-profile.toml schema with an [attribution] section:

[attribution]
submitter = "@username"
notes = "Tested with 1-hour continuous dictation session, no crashes."

Contribution flow (manual, honour-system MVP)

1. User runs kon-bench on their hardware.
2. User runs kon-bench --compare against baseline to confirm improvement isn't noise.
3. User forks kon-configs, commits their TOML under configs/<vendor>/, opens PR.
4. Maintainer reviews format + plausibility, merges.
5. No CI replay — revisit if spam becomes a problem.

Kon integration

• New Tauri command fetch_community_configs(gpu_fingerprint) — HTTPS GET https://raw.githubusercontent.com/<org>/kon-configs/main/index.json for the manifest, then fetches matching TOMLs
• Fingerprint match: GPU name substring + VRAM tier (e.g., "RTX 3060" + "12gb")
• Settings "Browse community configs" button lists matches with submitter, claimed RTF improvement, and a preview of the toggle deltas
• Applying a config updates Phase 1 toggles AND stores provenance (source = "community", submitter, fetch date)

What we explicitly skip at MVP

• No CI replay. Maintainer eyeballs + honour system. Revisit past ~50 configs or on abuse.
• No automated upload from kon-bench. User always commits + PRs manually. Zero privacy concerns, zero spam surface.
• No sophisticated fingerprint normalisation. Substring matching is sufficient.

Acceptance

• Repo exists with README + CONTRIBUTING + 2–3 seed configs
• Kon Settings fetches + lists + applies a community config end-to-end
• "Revert to default" path works (Phase 1's reset)

---

User experience — the one-click path

This is the UX the three phases together enable. All three are prerequisites; Phase 3-lite is what turns "run a CLI" into "click a button."

First-launch onboarding nudge

After the existing first-run model download, Kon surfaces a non-modal card:

🎛  GPU Optimisation
Detected: NVIDIA RTX 4070 (12 GB) · Linux Wayland
Current: Default GGML kernels

[ Auto-optimise ]   [ Show advanced ]   [ Skip ]

"Auto-optimise" triggers the hybrid flow below. "Show advanced" expands the Phase 1 toggle panel directly. "Skip" dismisses; user can always come back via Settings.

The "Auto-optimise" flow

Two steps, in this order:

Step 1 — Community config check (instant, ~2 s)

Kon fingerprints the GPU and queries the kon-configs manifest for matches. If a match exists, a preview card appears:

┌─────────────────────────────────────────────┐
│ Community config available                  │
│                                             │
│ From:     @someuser                         │
│ Claimed:  27% faster · 0% accuracy drift    │
│ Tested:   2026-04-21, driver nvidia 550     │
│                                             │
│ Changes 2 settings:                         │
│   • Cooperative matrix:  on → off           │
│   • Integer dot product: on → off           │
│                                             │
│ [ Apply (restart required) ]   [ Cancel ]   │
└─────────────────────────────────────────────┘

Apply → settings persist → restart prompt → done. 15 seconds end-to-end.

Step 2 — Fallback to local benchmark

If no community match, or the user prefers their own measurement:

┌─────────────────────────────────────────────┐
│ No community config for your hardware yet   │
│                                             │
│ We can benchmark your machine to find the   │
│ best settings. Takes ~8 minutes; runs in    │
│ the background while you keep using Kon.    │
│                                             │
│ [ Benchmark my GPU ]   [ Skip ]             │
└─────────────────────────────────────────────┘

Kicks off kon-bench as a background process. Kon keeps working during the run.

Progress UI during benchmark

Non-modal. Status chip in the lower-right of the main window:

⚙  Benchmarking GPU · 4 of 12 tested · ~5 min remaining   [ cancel ]

On completion, a toast:

Your GPU is 27% faster with the new config.   [ Review → ]

Review opens the same preview card as the community-config flow, with the same Apply / Cancel options.

After applying

Settings shows the active config's provenance:

• Using community config · applied 2026-04-21 · by @someuser
• Using auto-tuned config · benchmarked 2026-04-21
• Using defaults

Plus a "Revert to previous config" button, active for 7 days after any change, in case the new config misbehaves in real use (silent accuracy drift, crashes on long sessions, etc.) that the benchmark didn't catch.

Optional — sharing back to the community

After a successful local benchmark that shows meaningful gains, Kon prompts:

┌─────────────────────────────────────────────┐
│ Share your config with the community?       │
│                                             │
│ Your RTX 4070 tuning got you 27% faster.    │
│ Other RTX 4070 users would benefit.         │
│                                             │
│ Shared data: GPU name, driver version, OS,  │
│ config flags, benchmark numbers.            │
│ NOT shared: personal info, audio, anything  │
│ that identifies you beyond the GitHub fork. │
│                                             │
│ [ Review payload ]   [ Create PR ]   [ No ] │
└─────────────────────────────────────────────┘

"Create PR" opens the user's browser to github.com/…/kon-configs/new/main with the TOML prefilled in the PR body. User finishes the submission on GitHub (still honour-system; no automated uploads, no telemetry).

Non-GPU / integrated-only fallback

If sysinfo reports no dedicated GPU or Vulkan isn't available, the card replaces itself with:

🎛  GPU Optimisation
No dedicated GPU detected — Kon is using CPU inference.
GPU tuning doesn't apply to this setup.

No nag, no hidden settings, no broken experience.

Yes, "one click" is achievable

For users whose GPU has a community-contributed config, the experience is literally one click (the Apply button), plus a restart. ~15 seconds.

For users without a community match, the experience is two clicks (trigger bench → apply results on completion), with a passive ~8-minute background wait in between.

For users on integrated graphics / no GPU, the experience is zero clicks — Kon tells them GPU tuning doesn't apply and moves on.

---

Sequencing

Strict linear: Phase 1 → Phase 2 → Phase 3-lite. Each phase merges to main and gets dogfooded before the next starts.

• Phase 1 is a prereq for Phase 2 — kon-bench's output needs the Phase 1 settings schema to be its consumption target.
• Phase 2 is a prereq for Phase 3-lite — the community repo's config TOML schema is Phase 2's output schema (with an added [attribution] section).

Shelved with rationale

• Phase 4 — custom SPIR-V shader drops. Blocked on ggml-dedup workstream. Pinned in memory.
• Phase 5 — agentic (Karpathy-style) autotune. Phase 2's grid search produces schema-compatible results, so Phase 5 can drop in later without a schema break. Pinned.
• Phase 3's CI replay. Defer until spam / bad-config abuse is a real problem rather than a hypothetical one. Honour-system PR review is sufficient for the MVP community.
• kon-bench automated upload. Deliberately manual for MVP — removes all privacy / spam / rate-limiting concerns. Revisit when the community volume justifies the infrastructure.