jake/Lumotia
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: battery- and GPU-aware inference thread tuning design

Browse Source 
Spec for two new clamps on the existing inference_thread_count helper:
- Battery clamp: drop to physical/2 when on battery (Linux sysfs probe;
  macOS/Windows return Unknown, treated as OnAc).
- GPU-offload clamp: 2 threads for fully-offloaded LLM, 4 for Whisper
  (Whisper keeps mel spectrogram, decoder bookkeeping, and beam search
  on CPU even with full Vulkan offload).

Codex + online-research consult: no prior art in llama.cpp, Ollama,
Jan, or mistral.rs. Apple's Low Power Mode is OS-level (P-core
frequency cap), not a published software API. Lumenote would be first
in the open Rust inference-wrapper space; instrumentation matters,
hence the thread_sweep.rs extension + per-process INFO log.

Architecture (Approach B):
- New crates/core/src/power.rs (PowerState, sysfs probe, 10s TTL).
- New crates/core/src/tuning.rs (Workload enum, helper).
- Move vulkan_loader_available() from src-tauri to magnotia-core
  so crates/transcription can call it without a Tauri dep.
- Existing constants::inference_thread_count() becomes deprecated
  facade for one commit, then removed.

GPU-offload detection is intent-based (use_gpu && requested_layers
>= model.n_layer() for LLM; cfg(whisper-vulkan) && libvulkan
resolvable for Whisper). Residency-based detection deferred:
llama-cpp-2 0.1.145 doesn't expose post-load offload count, so a
true outcome flag would need llama_log_set callback parsing or an
upstream PR. Caveat documented in the spec.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jars
2026-05-09 10:36:21 +01:00
parent 4cb954ece4
commit 200c4fb447
1 changed files with 232 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

232
docs/superpowers/specs/2026-05-09-battery-gpu-aware-thread-tuning-design.md Normal file
View File

@@ -0,0 +1,232 @@
---
name: Battery- and GPU-aware inference thread tuning
type: spec
description: Power-aware and GPU-offload-aware clamps on inference_thread_count for whisper.cpp + llama.cpp call sites
date: 2026-05-09
status: draft
author: jars
---
# Battery- and GPU-aware inference thread tuning
## Problem
`crates/core/src/constants.rs::inference_thread_count()` currently returns a fixed clamp `[2, 8]` of physical-core count, regardless of:
1. Whether the laptop is on AC or battery — running 6 inference threads on battery wastes ~50% package power for ~30% extra throughput, on a workload the user can usually wait an extra second for.
2. Whether the model is fully offloaded to the GPU — when llama.cpp has all layers on Vulkan, the CPU does negligible matmul, and 6 worker threads is roughly 4 more than necessary; the same applies less aggressively to whisper.cpp.
Two new clamps wanted:
- **Battery clamp** — when on battery, drop to `physical / 2`. Apple's published Low Power Mode trade is roughly "30% perf for 50% package power" at the OS level; we replicate that pattern at the inference-thread layer where it actually matters for our workload.
- **GPU-offload clamp** — when fully offloaded, drop threads to a workload-specific floor (LLM: 2, Whisper: 4).
Floors differ because the workloads differ. llama.cpp does near-zero CPU matmul once layers are on the GPU; whisper.cpp keeps mel spectrogram, decoder bookkeeping, and beam search on the CPU regardless.
## Prior art
None observed. Codex investigation found no battery-aware thread scaling in llama.cpp, Ollama, Jan, or mistral.rs. LM Studio's internals are not public so it cannot be ruled out, but the pattern appears to be genuinely bespoke. Lumenote would be first in the open Rust inference-wrapper space.
The Apple "30% perf for 50% power" framing is empirical, not a published software API — Low Power Mode is OS-level (P-core frequency cap, background priority reduction). We are inventing the heuristic for our layer; instrumentation matters.
## Architecture
### Files
- **New** `crates/core/src/power.rs` — `PowerState`, sysfs probe, TTL cache.
- **New** `crates/core/src/tuning.rs` — `Workload`, `inference_thread_count(workload, gpu_offloaded)`.
- **Move** `vulkan_loader_available()` from `src-tauri/src/commands/models.rs` to `magnotia_core::hardware`. `crates/transcription` needs to call it; src-tauri continues to call it from its new home. `libloading = "0.8"` becomes a workspace dependency of `magnotia-core`.
- **Update** `crates/transcription/src/whisper_rs_backend.rs` — compute `gpu_offloaded`, pass to helper.
- **Update** `crates/llm/src/lib.rs` — compute `gpu_offloaded`, pass to helper. (Recompute *after* the model loads, since `model.n_layer()` requires the loaded model.)
- **Deprecate facade** `crates/core/src/constants.rs::inference_thread_count()` — keep as `#[deprecated]` re-export pointing at `tuning::inference_thread_count` for one commit, then remove.
### `power.rs`
```rust
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum PowerState { OnAc, OnBattery, Unknown }
pub fn probe_power_state() -> PowerState;
```
**Linux** (`cfg(target_os = "linux")`): walk `/sys/class/power_supply/`. For each entry, read `type` and `online`:
- If any entry has `type ∈ {"Mains", "USB"}` and `online == 1` → `OnAc`.
- Else if any entry has `type == "Battery"` → `OnBattery`.
- Else → `Unknown`.
`USB` is included because USB-C power delivery shows up as `type=USB` on modern laptops (e.g. Framework, recent ThinkPads). Mains is the conventional barrel-jack/wall-wart entry.
**macOS / Windows / other**: return `Unknown`. Native probes (`IOPSGetProvidingPowerSourceType`, `GetSystemPowerStatus`) are deferred to a follow-up.
**TTL cache**: `OnceLock<Mutex<(PowerState, Instant)>>` with 10s TTL. Re-probe on expiry. Battery state changes slowly; 10s is far below human-perceptible latency for thread-count adjustment.
**Test override**: `MAGNOTIA_POWER_STATE_OVERRIDE=ac|battery|unknown` short-circuits the probe. Always-on (not just `cfg(test)`) so `thread_sweep.rs` can drive both states without unplugging hardware. Documented as test-only; users overriding it in production is benign.
### `tuning.rs`
```rust
#[derive(Debug, Clone, Copy)]
pub enum Workload {
/// Llama-style transformer LLM. Near-zero CPU work when fully
/// offloaded; CPU floor in that case is GPU_FLOOR_LLM.
Llm,
/// Whisper transcription. Keeps mel spectrogram, decoder
/// bookkeeping, and beam search on the CPU even with full Vulkan
/// offload, so the CPU floor is higher: GPU_FLOOR_WHISPER.
Whisper,
}
const GPU_FLOOR_LLM: usize = 2;
const GPU_FLOOR_WHISPER: usize = 4;
pub fn inference_thread_count(workload: Workload, gpu_offloaded: bool) -> usize {
// 1. MAGNOTIA_INFERENCE_THREADS env override → absolute bypass
if let Ok(s) = std::env::var("MAGNOTIA_INFERENCE_THREADS") {
if let Ok(n) = s.parse::<usize>() {
if n > 0 { return n; }
}
}
// 2. base = physical cores (fallback: available_parallelism)
let physical = num_cpus::get_physical();
let mut chosen = if physical > 0 {
physical
} else {
std::thread::available_parallelism()
.map(|p| p.get())
.unwrap_or(MIN_INFERENCE_THREADS)
};
// 3. Battery clamp
if power::probe_power_state() == PowerState::OnBattery {
chosen = chosen / 2;
}
// 4. GPU-offload clamp
if gpu_offloaded {
let floor = match workload {
Workload::Llm => GPU_FLOOR_LLM,
Workload::Whisper => GPU_FLOOR_WHISPER,
};
chosen = chosen.min(floor);
}
// 5. Final clamp
chosen.clamp(MIN_INFERENCE_THREADS, MAX_INFERENCE_THREADS)
}
```
`PowerState::Unknown` is treated as `OnAc` (no clamp). Fail-open to existing behaviour on platforms we don't probe yet.
### Per-machine truth table
| Machine | AC+CPU | Battery+CPU | AC+GPU LLM | AC+GPU Whisper | Bat+GPU LLM | Bat+GPU Whisper |
|---|---|---|---|---|---|---|
| 4c4t budget | 4 | 2 | 2 | 4 | 2 | 2 |
| 6c12t (Ryzen 4650U) | 6 | 3 | 2 | 4 | 2 | 3 |
| 12c24t big-iron | 8 | 6 | 2 | 4 | 2 | 4 |
| 1c container/VM | 2 | 2 | 2 | 2 | 2 | 2 |
Algebra check: battery on 6c → 6/2 = 3, no further GPU clamp on whisper because 3 < 4 already. On 4c → 4/2 = 2, hits MIN floor. On 1c → fallback to available_parallelism; clamped up to MIN=2.
### GPU-offload detection
**LLM** (`crates/llm/src/lib.rs`):
```rust
let gpu_offloaded = use_gpu
&& i32::try_from(model.n_layer())
.map(|n| requested_n_gpu_layers >= n)
.unwrap_or(false);
```
`use_gpu` is the caller intent (already in `LoadedModelState`). `requested_n_gpu_layers` is what we passed into `LlamaModelParams::with_n_gpu_layers` — currently `u32::MAX` when `use_gpu`. Cross-checking against `model.n_layer()` (queryable post-load via `LlamaModel::n_layer()`) catches the trivially-true case while staying intent-based; if upstream changes the "max" sentinel semantics, the cross-check stays correct.
**Caveat documented in code**: residency is not actually verified. If VRAM is tight, llama.cpp can silently partial-offload and we'll under-thread the CPU. Real fix is a `llama_log_set` callback parsing `offloaded N/M layers` from llama.cpp's log output, or upstream PR to `llama-cpp-2` for a `n_gpu_layers_loaded()` getter (the underlying `llama_model_n_layer` is exposed but not the offload count). Tracked as observability follow-up; not blocking v1.
**Whisper** (`crates/transcription/src/whisper_rs_backend.rs`):
```rust
let gpu_offloaded = cfg!(feature = "whisper-vulkan")
&& magnotia_core::hardware::vulkan_loader_available();
```
Compile-time feature check (`whisper-vulkan` is in default features) plus runtime libvulkan resolution. If libvulkan1 isn't installed, whisper-rs falls back to CPU and we should not under-thread.
### Override behaviour
- `MAGNOTIA_INFERENCE_THREADS=N` (existing, unchanged) — absolute bypass. Returns N regardless of power/GPU state. For benchmarking, big-iron desktops, and users who want to opt out.
- `MAGNOTIA_POWER_STATE_OVERRIDE` — test only.
- No fine-grained `MAGNOTIA_BATTERY_CLAMP=off` or similar in v1. Add only if real users hit a case where the existing all-or-nothing override is insufficient.
## Tests
### Existing test extended
`crates/transcription/tests/thread_sweep.rs` already runs the JFK clip across n_threads values and prints an RTF table. Extend to four blocks driven by `MAGNOTIA_POWER_STATE_OVERRIDE`:
```
=== n_threads scaling: AC, CPU ===
=== n_threads scaling: AC, GPU (Vulkan) ===
=== n_threads scaling: battery, CPU ===
=== n_threads scaling: battery, GPU (Vulkan) ===
```
Each block uses the same `n_threads` sweep `[1, 2, 4, 6, 8, 12]` and reports xc_time, RTF, speedup_vs_1. This is the empirical instrument for validating the heuristic — if the GPU+battery whisper case shows the floor is wrong, the table tells us by how much.
### New unit tests
`crates/core/src/tuning.rs`:
- `battery_clamp_halves_on_battery` — set override to battery, assert returned value is roughly half of AC.
- `gpu_floor_lower_for_llm_than_whisper` — both with `gpu_offloaded=true`, assert Llm ≤ Whisper.
- `env_var_bypasses_all_clamps` — set `MAGNOTIA_INFERENCE_THREADS=10`, set battery, set gpu_offloaded; assert returns 10.
- `clamp_to_min_when_low_core_count` — fall through to min on 1-2 logical cores via available_parallelism (use a mock or just verify clamp inside the function).
- `clamp_to_max_for_huge_machines` — assert MAX_INFERENCE_THREADS=8 still binds.
`crates/core/src/power.rs`:
- `linux_sysfs_parses_ac_online` — write fixture entries to `tempdir`, parse, expect OnAc.
- `linux_sysfs_parses_battery_only` — fixture with only `type=Battery online=0`, expect OnBattery.
- `linux_sysfs_parses_unknown_when_empty` — empty dir, expect Unknown.
- `usb_pd_treated_as_ac` — fixture `type=USB online=1`, expect OnAc.
- `override_short_circuits_probe` — set env var, expect override value regardless of fixture.
- `non_linux_returns_unknown` — cfg-gated, macos/windows return Unknown.
- `ttl_cache_returns_same_value_within_window` — call once with fixture A, mutate fixture to B, call again immediately, assert second call returns A. Then advance time past TTL (test-only `force_expire()` in `power.rs`), call again, assert returns B.
For sysfs fixture tests, factor the parser into `parse_power_state_from_dir(path: &Path) -> PowerState` so tests can point it at a `tempdir`; the public `probe_power_state()` reads `/sys/class/power_supply/` by default.
## Logging
One INFO-level line on first call per process per (workload, on_battery, gpu_offloaded) combination. Cached in a `OnceLock<Mutex<HashSet<...>>>` keyed by the tuple to avoid log spam.
```
inference_thread_count: 3 workload=Whisper gpu_offloaded=false on_battery=true clamps=[battery]
```
`clamps=[]` when nothing fired beyond the base + final clamp, `[battery]` if battery halved, `[gpu]` if GPU floor bound, `[battery,gpu]` if both. This is the empirical instrument for users running real Lumenote sessions — gives Jake real data on what the heuristic does in the wild before we tune the constants.
## Out of scope (follow-ups)
- True LLM offload-residency observability via `llama_log_set` callback parsing or upstream PR to llama-cpp-2.
- macOS native probe via `IOPSGetProvidingPowerSourceType`.
- Windows native probe via `GetSystemPowerStatus`.
- UI affordance — "Running on battery — reduced AI performance" indicator next to the model badge.
- Power-state change events — current TTL polling is sufficient at our cadence; no need for a watcher thread.
- Lumenote rebrand of `MAGNOTIA_*` env vars — separate workstream coordinated with the broader rename.
## Risk and rollback
- **Heuristic is wrong** — telemetry log lets us see real impact. If the GPU+battery whisper floor of 4 is too aggressive (transcription stutters), bump it. The truth table is data we can update; the constants live in one file.
- **Sysfs probe panics on a non-standard distro** — wrapped in a graceful fallthrough to `Unknown`, never returns a Result-bubble. Worst case: clamp doesn't fire on that distro.
- **Vulkan loader probe regresses** — moving `vulkan_loader_available()` into core means src-tauri callers route through core. Wire-test via the existing `models.rs` accelerator-list code path; if accelerator-list still reports Vulkan correctly, the move worked.
- **Rollback** — single `git revert` on the migration commit returns both call sites to the old `inference_thread_count()` (preserved as deprecated facade for one commit). The deprecation-removal commit is a separate target.
## Acceptance
- All four panels of `thread_sweep.rs` print without panic on a 6c12t Ryzen.
- New unit tests pass on Linux (and on macOS CI where applicable; sysfs tests cfg-gated to Linux).
- Existing `cargo test -p magnotia-core` and `-p magnotia-llm` continue to pass.
- A real Lumenote session with on-battery laptop logs `on_battery=true clamps=[battery]` at first inference call.
- Manual smoke: kick a Whisper Tiny transcription on battery, verify CPU usage settles around half what it was on AC.