Lumotia/docs/superpowers/specs/2026-05-09-battery-gpu-aware-thread-tuning-design.md

name, type, description, date, status, author

name	type	description	date	status	author
Battery- and GPU-aware inference thread tuning	spec	Power-aware and GPU-offload-aware clamps on inference_thread_count for whisper.cpp + llama.cpp call sites	2026-05-09	draft	jars

Battery- and GPU-aware inference thread tuning

Problem

crates/core/src/constants.rs::inference_thread_count() currently returns a fixed clamp [2, 8] of physical-core count, regardless of:

1. Whether the laptop is on AC or battery — running 6 inference threads on battery wastes ~50% package power for ~30% extra throughput, on a workload the user can usually wait an extra second for.
2. Whether the model is fully offloaded to the GPU — when llama.cpp has all layers on Vulkan, the CPU does negligible matmul, and 6 worker threads is roughly 4 more than necessary; the same applies less aggressively to whisper.cpp.

Two new clamps wanted:

• Battery clamp — when on battery, drop to physical / 2. Apple's published Low Power Mode trade is roughly "30% perf for 50% package power" at the OS level; we replicate that pattern at the inference-thread layer where it actually matters for our workload.
• GPU-offload clamp — when fully offloaded, drop threads to a workload-specific floor (LLM: 2, Whisper: 4).

Floors differ because the workloads differ. llama.cpp does near-zero CPU matmul once layers are on the GPU; whisper.cpp keeps mel spectrogram, decoder bookkeeping, and beam search on the CPU regardless.

Prior art

None observed. Codex investigation found no battery-aware thread scaling in llama.cpp, Ollama, Jan, or mistral.rs. LM Studio's internals are not public so it cannot be ruled out, but the pattern appears to be genuinely bespoke. Lumenote would be first in the open Rust inference-wrapper space.

The Apple "30% perf for 50% power" framing is empirical, not a published software API — Low Power Mode is OS-level (P-core frequency cap, background priority reduction). We are inventing the heuristic for our layer; instrumentation matters.

Architecture

Files

• New crates/core/src/power.rs — PowerState, sysfs probe, TTL cache.
• New crates/core/src/tuning.rs — Workload, inference_thread_count(workload, gpu_offloaded).
• Move vulkan_loader_available() from src-tauri/src/commands/models.rs to lumotia_core::hardware. crates/transcription needs to call it; src-tauri continues to call it from its new home. libloading = "0.8" becomes a workspace dependency of lumotia-core.
• Update crates/transcription/src/whisper_rs_backend.rs — compute gpu_offloaded, pass to helper.
• Update crates/llm/src/lib.rs — compute gpu_offloaded, pass to helper. (Recompute after the model loads, since model.n_layer() requires the loaded model.)
• Deprecate facade crates/core/src/constants.rs::inference_thread_count() — keep as #[deprecated] re-export pointing at tuning::inference_thread_count for one commit, then remove.

power.rs

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum PowerState { OnAc, OnBattery, Unknown }

pub fn probe_power_state() -> PowerState;

Linux (cfg(target_os = "linux")): walk /sys/class/power_supply/. For each entry, read type and online:

• If any entry has type ∈ {"Mains", "USB"} and online == 1 → OnAc.
• Else if any entry has type == "Battery" → OnBattery.
• Else → Unknown.

USB is included because USB-C power delivery shows up as type=USB on modern laptops (e.g. Framework, recent ThinkPads). Mains is the conventional barrel-jack/wall-wart entry.

macOS / Windows / other: return Unknown. Native probes (IOPSGetProvidingPowerSourceType, GetSystemPowerStatus) are deferred to a follow-up.

TTL cache: OnceLock<Mutex<(PowerState, Instant)>> with 10s TTL. Re-probe on expiry. Battery state changes slowly; 10s is far below human-perceptible latency for thread-count adjustment.

Test override: LUMOTIA_POWER_STATE_OVERRIDE=ac|battery|unknown short-circuits the probe. Always-on (not just cfg(test)) so thread_sweep.rs can drive both states without unplugging hardware. Documented as test-only; users overriding it in production is benign.

tuning.rs

#[derive(Debug, Clone, Copy)]
pub enum Workload {
    /// Llama-style transformer LLM. Near-zero CPU work when fully
    /// offloaded; CPU floor in that case is GPU_FLOOR_LLM.
    Llm,
    /// Whisper transcription. Keeps mel spectrogram, decoder
    /// bookkeeping, and beam search on the CPU even with full Vulkan
    /// offload, so the CPU floor is higher: GPU_FLOOR_WHISPER.
    Whisper,
}

const GPU_FLOOR_LLM: usize = 2;
const GPU_FLOOR_WHISPER: usize = 4;

pub fn inference_thread_count(workload: Workload, gpu_offloaded: bool) -> usize {
    // 1. LUMOTIA_INFERENCE_THREADS env override → absolute bypass
    if let Ok(s) = std::env::var("LUMOTIA_INFERENCE_THREADS") {
        if let Ok(n) = s.parse::<usize>() {
            if n > 0 { return n; }
        }
    }

    // 2. base = physical cores (fallback: available_parallelism)
    let physical = num_cpus::get_physical();
    let mut chosen = if physical > 0 {
        physical
    } else {
        std::thread::available_parallelism()
            .map(|p| p.get())
            .unwrap_or(MIN_INFERENCE_THREADS)
    };

    // 3. Battery clamp
    if power::probe_power_state() == PowerState::OnBattery {
        chosen = chosen / 2;
    }

    // 4. GPU-offload clamp
    if gpu_offloaded {
        let floor = match workload {
            Workload::Llm => GPU_FLOOR_LLM,
            Workload::Whisper => GPU_FLOOR_WHISPER,
        };
        chosen = chosen.min(floor);
    }

    // 5. Final clamp
    chosen.clamp(MIN_INFERENCE_THREADS, MAX_INFERENCE_THREADS)
}

PowerState::Unknown is treated as OnAc (no clamp). Fail-open to existing behaviour on platforms we don't probe yet.

Per-machine truth table

Machine	AC+CPU	Battery+CPU	AC+GPU LLM	AC+GPU Whisper	Bat+GPU LLM	Bat+GPU Whisper
4c4t budget	4	2	2	4	2	2
6c12t (Ryzen 4650U)	6	3	2	4	2	3
12c24t big-iron	8	6	2	4	2	4
1c container/VM	2	2	2	2	2	2

Algebra check: battery on 6c → 6/2 = 3, no further GPU clamp on whisper because 3 < 4 already. On 4c → 4/2 = 2, hits MIN floor. On 1c → fallback to available_parallelism; clamped up to MIN=2.

GPU-offload detection

LLM (crates/llm/src/lib.rs):

let gpu_offloaded = use_gpu
    && i32::try_from(model.n_layer())
        .map(|n| requested_n_gpu_layers >= n)
        .unwrap_or(false);

use_gpu is the caller intent (already in LoadedModelState). requested_n_gpu_layers is what we passed into LlamaModelParams::with_n_gpu_layers — currently u32::MAX when use_gpu. Cross-checking against model.n_layer() (queryable post-load via LlamaModel::n_layer()) catches the trivially-true case while staying intent-based; if upstream changes the "max" sentinel semantics, the cross-check stays correct.

Caveat documented in code: residency is not actually verified. If VRAM is tight, llama.cpp can silently partial-offload and we'll under-thread the CPU. Real fix is a llama_log_set callback parsing offloaded N/M layers from llama.cpp's log output, or upstream PR to llama-cpp-2 for a n_gpu_layers_loaded() getter (the underlying llama_model_n_layer is exposed but not the offload count). Tracked as observability follow-up; not blocking v1.

Whisper (crates/transcription/src/whisper_rs_backend.rs):

let gpu_offloaded = cfg!(feature = "whisper-vulkan")
    && lumotia_core::hardware::vulkan_loader_available();

Compile-time feature check (whisper-vulkan is in default features) plus runtime libvulkan resolution. If libvulkan1 isn't installed, whisper-rs falls back to CPU and we should not under-thread.

Override behaviour

• LUMOTIA_INFERENCE_THREADS=N (existing, unchanged) — absolute bypass. Returns N regardless of power/GPU state. For benchmarking, big-iron desktops, and users who want to opt out.
• LUMOTIA_POWER_STATE_OVERRIDE — test only.
• No fine-grained MAGNOTIA_BATTERY_CLAMP=off or similar in v1. Add only if real users hit a case where the existing all-or-nothing override is insufficient.

Tests

Existing test extended

crates/transcription/tests/thread_sweep.rs already runs the JFK clip across n_threads values and prints an RTF table. Extend to four blocks driven by LUMOTIA_POWER_STATE_OVERRIDE:

=== n_threads scaling: AC, CPU ===
=== n_threads scaling: AC, GPU (Vulkan) ===
=== n_threads scaling: battery, CPU ===
=== n_threads scaling: battery, GPU (Vulkan) ===

Each block uses the same n_threads sweep [1, 2, 4, 6, 8, 12] and reports xc_time, RTF, speedup_vs_1. This is the empirical instrument for validating the heuristic — if the GPU+battery whisper case shows the floor is wrong, the table tells us by how much.

New unit tests

crates/core/src/tuning.rs:

• battery_clamp_halves_on_battery — set override to battery, assert returned value is roughly half of AC.
• gpu_floor_lower_for_llm_than_whisper — both with gpu_offloaded=true, assert Llm ≤ Whisper.
• env_var_bypasses_all_clamps — set LUMOTIA_INFERENCE_THREADS=10, set battery, set gpu_offloaded; assert returns 10.
• clamp_to_min_when_low_core_count — fall through to min on 1-2 logical cores via available_parallelism (use a mock or just verify clamp inside the function).
• clamp_to_max_for_huge_machines — assert MAX_INFERENCE_THREADS=8 still binds.

crates/core/src/power.rs:

• linux_sysfs_parses_ac_online — write fixture entries to tempdir, parse, expect OnAc.
• linux_sysfs_parses_battery_only — fixture with only type=Battery online=0, expect OnBattery.
• linux_sysfs_parses_unknown_when_empty — empty dir, expect Unknown.
• usb_pd_treated_as_ac — fixture type=USB online=1, expect OnAc.
• override_short_circuits_probe — set env var, expect override value regardless of fixture.
• non_linux_returns_unknown — cfg-gated, macos/windows return Unknown.
• ttl_cache_returns_same_value_within_window — call once with fixture A, mutate fixture to B, call again immediately, assert second call returns A. Then advance time past TTL (test-only force_expire() in power.rs), call again, assert returns B.

For sysfs fixture tests, factor the parser into parse_power_state_from_dir(path: &Path) -> PowerState so tests can point it at a tempdir; the public probe_power_state() reads /sys/class/power_supply/ by default.

Logging

One INFO-level line on first call per process per (workload, on_battery, gpu_offloaded) combination. Cached in a OnceLock<Mutex<HashSet<...>>> keyed by the tuple to avoid log spam.

inference_thread_count: 3 workload=Whisper gpu_offloaded=false on_battery=true clamps=[battery]

clamps=[] when nothing fired beyond the base + final clamp, [battery] if battery halved, [gpu] if GPU floor bound, [battery,gpu] if both. This is the empirical instrument for users running real Lumenote sessions — gives Jake real data on what the heuristic does in the wild before we tune the constants.

Out of scope (follow-ups)

• True LLM offload-residency observability via llama_log_set callback parsing or upstream PR to llama-cpp-2.
• macOS native probe via IOPSGetProvidingPowerSourceType.
• Windows native probe via GetSystemPowerStatus.
• UI affordance — "Running on battery — reduced AI performance" indicator next to the model badge.
• Power-state change events — current TTL polling is sufficient at our cadence; no need for a watcher thread.
• Lumenote rebrand of MAGNOTIA_* env vars — separate workstream coordinated with the broader rename.

Risk and rollback

• Heuristic is wrong — telemetry log lets us see real impact. If the GPU+battery whisper floor of 4 is too aggressive (transcription stutters), bump it. The truth table is data we can update; the constants live in one file.
• Sysfs probe panics on a non-standard distro — wrapped in a graceful fallthrough to Unknown, never returns a Result-bubble. Worst case: clamp doesn't fire on that distro.
• Vulkan loader probe regresses — moving vulkan_loader_available() into core means src-tauri callers route through core. Wire-test via the existing models.rs accelerator-list code path; if accelerator-list still reports Vulkan correctly, the move worked.
• Rollback — single git revert on the migration commit returns both call sites to the old inference_thread_count() (preserved as deprecated facade for one commit). The deprecation-removal commit is a separate target.

Acceptance

• All four panels of thread_sweep.rs print without panic on a 6c12t Ryzen.
• New unit tests pass on Linux (and on macOS CI where applicable; sysfs tests cfg-gated to Linux).
• Existing cargo test -p lumotia-core and -p lumotia-llm continue to pass.
• A real Lumenote session with on-battery laptop logs on_battery=true clamps=[battery] at first inference call.
• Manual smoke: kick a Whisper Tiny transcription on battery, verify CPU usage settles around half what it was on AC.