Lumotia/docs/gpu-tuning/plan.md

# Magnotia — GPU Tuning & Community Config Plan

*Implementation spec for the first three phases of the GPU kernel tuning roadmap. The full five-phase roadmap is pinned in memory; this document scopes the MVP subset that ships real value without pulling in `ggml`-dedup or agentic-search prerequisites.*

## Scope

**IN** (this document):

- Phase 1 — Advanced GPU tuning settings panel (exposing GGML env vars)
- Phase 2 — `magnotia-bench` local autotuning CLI
- Phase 3-lite — `magnotia-configs` community repo with manual-PR workflow (no CI replay)

**OUT** (pinned to memory for later):

- Phase 4 — custom SPIR-V shader drops (blocked on `ggml`-dedup)
- Phase 5 — Karpathy-style agentic autotuning
- CI replay for community repo (defer until spam / bad configs become a real problem)

This subset captures roughly 85% of the perceived value for ~20% of the total effort. The deferred pieces are where complexity explodes; the MVP stops before it.

---

## Phase 1 — Advanced GPU tuning settings panel

**Effort**: 1–2 days.

**What ships**: Settings → Advanced → GPU Tuning collapsible section with toggles for GGML env vars. Env vars are applied at app startup before any GPU backend initialises. Per-profile storage; restart required to take effect.

### Toggles shipped at MVP

| UI label | Env var | Default | When users enable |
|---|---|---|---|
| Disable cooperative matrix | `GGML_VK_DISABLE_COOPMAT` | off | "Inference hangs" on RDNA2 / buggy Mesa versions |
| Force FP32 math | `GGML_VK_FORCE_FP32` | off | "Garbage transcripts" on Intel Arc / older NVIDIA |
| Disable FP16 ops | `GGML_VK_DISABLE_F16` | off | Silent-fail on some Mesa 22.x builds |
| Disable integer dot product | `GGML_VK_DISABLE_INTEGER_DOT_PRODUCT` | off | "Random NaN" on RDNA2 with certain drivers |
| Enable Vulkan validation | `GGML_VK_VALIDATE` | off | Diagnostic only; impacts performance |

Metal / CUDA counterparts slot in when those backends grow in Magnotia. Today Magnotia is Vulkan-only.

### Design

- New `SettingsState.gpuTuning: { disableCoopmat: boolean, forceFp32: boolean, disableF16: boolean, disableIntegerDotProduct: boolean, enableValidation: boolean }` in [src/lib/types/app.ts](../../src/lib/types/app.ts)
- All defaults `false` in [src/lib/stores/page.svelte.ts](../../src/lib/stores/page.svelte.ts)
- Persistence uses the existing `save_preferences` → SQLite `magnotia_preferences` path
- Backend process env must be established before GPU backend initialisation. **Superseded note (2026-05-12):** runtime `std::env::set_var` mutation and the old `ensure_x11_on_wayland` pattern were removed from `src-tauri/src/lib.rs`; launcher/wrapper processes now own env-var setup. Preserve this startup-order requirement when revisiting GPU tuning, but do not reintroduce in-process env mutation.
- Settings UI shows a sticky "Restart required for changes to take effect" banner when any toggle has drifted from its launch-time value
- A "Reset to defaults" button zeroes all toggles

### Acceptance

- Toggling "Disable cooperative matrix" on and restarting → `vulkaninfo` (or GGML debug logs) confirms the knob is honoured at backend init
- Default all-off produces identical performance + transcription output to the current `main` (smoke test)
- An integration test with a fake settings fixture confirms env vars are set before `AppState` initialises

---

## Phase 2 — `magnotia-bench` local autotuning CLI

**Effort**: 3–5 days.

**What ships**: New workspace binary `crates/bench/` producing a `magnotia-bench` executable. User runs it once post-install; output lands at `~/.magnotia/gpu-profile.toml` with the best-scoring config for their hardware. Settings page gets an "Apply auto-tuned profile" button that consumes the TOML and updates the Phase 1 toggles.

### CLI surface

```
magnotia-bench --quick                      # bundled 20s sample + reference transcript
magnotia-bench --model <path> --audio <wav> --transcript <txt>
magnotia-bench --compare <profile.toml>     # benchmark a specific profile vs default
```

### Execution model

Grid-search via **subprocess spawning**. Each config variant runs in a child process with its own env vars — because env vars must be set at process startup; you cannot safely mutate GGML's runtime state once it's initialised. The parent serialises variants, spawns a child per variant, waits for each to exit with a JSON line on stdout, aggregates and ranks.

### Search strategy (not naive combinatorial)

1. Run baseline (all defaults).
2. Run each single-flag variant against baseline.
3. Take the top-3 single flags by RTF improvement with zero WER drift.
4. Combine pairwise.
5. Top-scored composite config wins.

This gives us ~9–15 subprocess runs instead of the ~32 a full combinatorial sweep would need; converges on local optima without the combinatorial explosion.

### Metrics

- **Real-time factor (RTF)** = `audio_seconds / inference_wall_seconds`. Lower is better.
- **Word error rate (WER)** against the ground-truth transcript. Any config with >0.5% WER drift from baseline is rejected regardless of RTF improvement.
- **Peak VRAM** (optional, best-effort via `nvidia-smi` / `rocm-smi` sampling).

### Runtime

~5–15 minutes on typical hardware. Progress bar + ETA rendered to stderr so stdout stays machine-readable.

### Bundled fixture

A 20-second public-domain speech clip with a known-good reference transcript, committed to `crates/bench/fixtures/`. Source: LibriVox recording (CC0).

### Output schema (`gpu-profile.toml`)

```toml
[benchmarked_at]
timestamp = "2026-04-21T14:32:00Z"
magnotia_version = "0.1.0"
model = "whisper-distil-large-v3"

[hardware]
gpu_name = "NVIDIA GeForce RTX 4070"
vram_mb = 12282
driver = "nvidia 550.120"
os = "linux"
mesa = ""

[baseline]
rtf = 0.043
wer = 0.028

[best]
rtf = 0.031
rtf_improvement = 0.279    # 27.9% faster
wer = 0.028

[best.env]
GGML_VK_DISABLE_COOPMAT = "0"
GGML_VK_FORCE_FP32 = "0"
# … full flag set, including unchanged ones, for reproducibility
```

### Crate layout

```
crates/bench/
├── Cargo.toml
├── fixtures/
│   ├── librivox-sample.wav
│   └── librivox-sample.txt
└── src/
    ├── main.rs         # CLI + parent process
    ├── runner.rs       # subprocess harness (child entry gate: MAGNOTIA_BENCH_RUN=1)
    ├── matrix.rs       # grid-search + top-k logic
    ├── metrics.rs      # RTF + WER + optional VRAM sampling
    └── profile.rs      # TOML serialise
```

Depends on `magnotia-transcription` + `magnotia-llm` + `magnotia-audio` as path deps so it reuses the existing model-loading code.

### Acceptance

- `magnotia-bench --quick` runs unattended to completion on a fresh install
- Produces a valid `gpu-profile.toml`
- "Apply auto-tuned" button in Settings consumes the TOML and updates Phase 1 toggles (restart banner fires as expected)
- Re-running with `--compare <profile>` produces reproducible-enough numbers (RTF within 5% run-to-run)

---

## Phase 3-lite — `magnotia-configs` community repo

**Effort**: 3 days (1 for repo + seeds, 2 for Magnotia-side fetch + apply UI).

**What ships**: A separate public GitHub repo `magnotia-configs` (not part of the magnotia main repo) seeded with 2–3 curated configs. Magnotia's Settings page gets a "Browse community configs" button that fetches matching configs for the user's detected hardware.

### Repo structure

```
magnotia-configs/
├── README.md              # pitch + how to benefit
├── CONTRIBUTING.md        # required fields, benchmark protocol, fork/PR flow
├── SCHEMA.md              # TOML schema documentation
├── index.json             # manifest for Magnotia to discover configs
└── configs/
    ├── nvidia/
    │   ├── rtx-3060-12gb-linux.toml
    │   └── rtx-4070-linux.toml
    ├── amd/
    │   └── rx-6700xt-mesa-23-linux.toml
    └── intel/
        └── arc-a770-windows.toml
```

### Config TOML

Extends Phase 2's `gpu-profile.toml` schema with an `[attribution]` section:

```toml
[attribution]
submitter = "@username"
notes = "Tested with 1-hour continuous dictation session, no crashes."
```

### Contribution flow (manual, honour-system MVP)

1. User runs `magnotia-bench` on their hardware.
2. User runs `magnotia-bench --compare` against baseline to confirm improvement isn't noise.
3. User forks `magnotia-configs`, commits their TOML under `configs/<vendor>/`, opens PR.
4. Maintainer reviews format + plausibility, merges.
5. No CI replay — revisit if spam becomes a problem.

### Magnotia integration

- New Tauri command `fetch_community_configs(gpu_fingerprint)` — HTTPS GET `https://raw.githubusercontent.com/<org>/magnotia-configs/main/index.json` for the manifest, then fetches matching TOMLs
- Fingerprint match: GPU name substring + VRAM tier (e.g., `"RTX 3060"` + `"12gb"`)
- Settings "Browse community configs" button lists matches with submitter, claimed RTF improvement, and a preview of the toggle deltas
- Applying a config updates Phase 1 toggles AND stores provenance (source = `"community"`, submitter, fetch date)

### What we explicitly skip at MVP

- **No CI replay**. Maintainer eyeballs + honour system. Revisit past ~50 configs or on abuse.
- **No automated upload from `magnotia-bench`**. User always commits + PRs manually. Zero privacy concerns, zero spam surface.
- **No sophisticated fingerprint normalisation**. Substring matching is sufficient.

### Acceptance

- Repo exists with README + CONTRIBUTING + 2–3 seed configs
- Magnotia Settings fetches + lists + applies a community config end-to-end
- "Revert to default" path works (Phase 1's reset)

---

## User experience — the one-click path

This is the UX the three phases together enable. All three are prerequisites; Phase 3-lite is what turns "run a CLI" into "click a button."

### First-launch onboarding nudge

After the existing first-run model download, Magnotia surfaces a non-modal card:

```
🎛  GPU Optimisation
Detected: NVIDIA RTX 4070 (12 GB) · Linux Wayland
Current: Default GGML kernels

[ Auto-optimise ]   [ Show advanced ]   [ Skip ]
```

"Auto-optimise" triggers the hybrid flow below. "Show advanced" expands the Phase 1 toggle panel directly. "Skip" dismisses; user can always come back via Settings.

### The "Auto-optimise" flow

Two steps, in this order:

**Step 1 — Community config check (instant, ~2 s)**

Magnotia fingerprints the GPU and queries the `magnotia-configs` manifest for matches. If a match exists, a preview card appears:

```
┌─────────────────────────────────────────────┐
│ Community config available                  │
│                                             │
│ From:     @someuser                         │
│ Claimed:  27% faster · 0% accuracy drift    │
│ Tested:   2026-04-21, driver nvidia 550     │
│                                             │
│ Changes 2 settings:                         │
│   • Cooperative matrix:  on → off           │
│   • Integer dot product: on → off           │
│                                             │
│ [ Apply (restart required) ]   [ Cancel ]   │
└─────────────────────────────────────────────┘
```

Apply → settings persist → restart prompt → done. 15 seconds end-to-end.

**Step 2 — Fallback to local benchmark**

If no community match, or the user prefers their own measurement:

```
┌─────────────────────────────────────────────┐
│ No community config for your hardware yet   │
│                                             │
│ We can benchmark your machine to find the   │
│ best settings. Takes ~8 minutes; runs in    │
│ the background while you keep using Magnotia.    │
│                                             │
│ [ Benchmark my GPU ]   [ Skip ]             │
└─────────────────────────────────────────────┘
```

Kicks off `magnotia-bench` as a background process. Magnotia keeps working during the run.

### Progress UI during benchmark

Non-modal. Status chip in the lower-right of the main window:

```
⚙  Benchmarking GPU · 4 of 12 tested · ~5 min remaining   [ cancel ]
```

On completion, a toast:

```
Your GPU is 27% faster with the new config.   [ Review → ]
```

Review opens the same preview card as the community-config flow, with the same Apply / Cancel options.

### After applying

Settings shows the active config's provenance:

- `Using community config · applied 2026-04-21 · by @someuser`
- `Using auto-tuned config · benchmarked 2026-04-21`
- `Using defaults`

Plus a "Revert to previous config" button, active for 7 days after any change, in case the new config misbehaves in real use (silent accuracy drift, crashes on long sessions, etc.) that the benchmark didn't catch.

### Optional — sharing back to the community

After a successful local benchmark that shows meaningful gains, Magnotia prompts:

```
┌─────────────────────────────────────────────┐
│ Share your config with the community?       │
│                                             │
│ Your RTX 4070 tuning got you 27% faster.    │
│ Other RTX 4070 users would benefit.         │
│                                             │
│ Shared data: GPU name, driver version, OS,  │
│ config flags, benchmark numbers.            │
│ NOT shared: personal info, audio, anything  │
│ that identifies you beyond the GitHub fork. │
│                                             │
│ [ Review payload ]   [ Create PR ]   [ No ] │
└─────────────────────────────────────────────┘
```

"Create PR" opens the user's browser to `github.com/…/magnotia-configs/new/main` with the TOML prefilled in the PR body. User finishes the submission on GitHub (still honour-system; no automated uploads, no telemetry).

### Non-GPU / integrated-only fallback

If `sysinfo` reports no dedicated GPU or Vulkan isn't available, the card replaces itself with:

```
🎛  GPU Optimisation
No dedicated GPU detected — Magnotia is using CPU inference.
GPU tuning doesn't apply to this setup.
```

No nag, no hidden settings, no broken experience.

### Yes, "one click" is achievable

For users whose GPU has a community-contributed config, the experience is **literally one click** (the Apply button), plus a restart. ~15 seconds.

For users without a community match, the experience is **two clicks** (trigger bench → apply results on completion), with a passive ~8-minute background wait in between.

For users on integrated graphics / no GPU, the experience is **zero clicks** — Magnotia tells them GPU tuning doesn't apply and moves on.

---

## Sequencing

Strict linear: Phase 1 → Phase 2 → Phase 3-lite. Each phase merges to `main` and gets dogfooded before the next starts.

- Phase 1 is a prereq for Phase 2 — `magnotia-bench`'s output needs the Phase 1 settings schema to be its consumption target.
- Phase 2 is a prereq for Phase 3-lite — the community repo's config TOML schema **is** Phase 2's output schema (with an added `[attribution]` section).

## Shelved with rationale

- **Phase 4 — custom SPIR-V shader drops.** Blocked on `ggml`-dedup workstream. Pinned in memory.
- **Phase 5 — agentic (Karpathy-style) autotune.** Phase 2's grid search produces schema-compatible results, so Phase 5 can drop in later without a schema break. Pinned.
- **Phase 3's CI replay.** Defer until spam / bad-config abuse is a real problem rather than a hypothetical one. Honour-system PR review is sufficient for the MVP community.
- **`magnotia-bench` automated upload.** Deliberately manual for MVP — removes all privacy / spam / rate-limiting concerns. Revisit when the community volume justifies the infrastructure.