Lumotia/docs/architecture-map/05-core-storage-hotkey-build/core-tuning.md

name, type, slice, last_verified

name	type	slice	last_verified
Core inference thread tuning	architecture-map-page	05-core-storage-hotkey-build	2026/05/09

Core inference thread tuning

Where you are: Architecture map → Core, Storage, Hotkey, Build → Core inference thread tuning

Plain English summary. A single helper, inference_thread_count, that answers "how many CPU threads should we give whisper.cpp / llama.cpp on this user's machine right now?" Combines the physical-core budget, battery awareness, and GPU-offload awareness into one number. Logs the chosen value once per (workload, battery, GPU-offload) tuple so the same configuration does not flood the journal.

At a glance

• File: crates/core/src/tuning.rs (233 LOC, 127 of which are tests).
• External deps: num_cpus 1, tracing 0.1, plus the in-crate power module.
• Public surface: MIN_INFERENCE_THREADS, MAX_INFERENCE_THREADS, Workload enum, inference_thread_count.
• Consumers: slice 3 (Whisper engine, Parakeet engine — Workload::Whisper); slice 4 (LLM engine — Workload::Llm).

What's in here

Constants — crates/core/src/tuning.rs:11, 20

pub const MIN_INFERENCE_THREADS: usize = 2;
pub const MAX_INFERENCE_THREADS: usize = 8;

Single-threaded inference is measurably worse than two on every multi-core part. Past 8 threads, whisper.cpp and llama.cpp scaling flattens because SMT siblings contend for the FPU during F16 / F32 matmul. 8 is a conservative ceiling that leaves <5% on the table for big-iron desktops while keeping consumer 6c/12t laptops out of contention territory.

Workload enum — crates/core/src/tuning.rs:23

pub enum Workload { Llm, Whisper }

The two workloads have different floors when GPU offload is on.

Internal floors — crates/core/src/tuning.rs:33-34

const GPU_FLOOR_LLM: usize = 2;
const GPU_FLOOR_WHISPER: usize = 4;

Whisper retains CPU work even with full Vulkan offload (mel spectrogram, decoder bookkeeping, beam search). LLM (llama-style transformer) drops to near-zero CPU work when fully offloaded. Architectural invariant tested at tuning.rs:200: GPU_FLOOR_WHISPER >= GPU_FLOOR_LLM.

inference_thread_count(workload, gpu_offloaded) -> usize — crates/core/src/tuning.rs:53

Resolution order:

1. MAGNOTIA_INFERENCE_THREADS=N — absolute bypass, returns N without any clamps.
2. Base = num_cpus::get_physical(). Falls back to std::thread::available_parallelism(), then to MIN_INFERENCE_THREADS = 2.
3. On battery → base /= 2. Power state is read from lumotia_core::power::probe_power_state(). The 10-second cache there means the call is cheap; see core-power.md.
4. GPU offloaded → clamp to GPU_FLOOR_<workload>. Whisper floor is 4, LLM floor is 2.
5. Final clamp to [MIN_INFERENCE_THREADS, MAX_INFERENCE_THREADS].

One-shot logging — crates/core/src/tuning.rs:39, 87-101

A static SEEN: OnceLock<Mutex<HashSet<(Workload, bool, bool)>>> records every distinct (workload, on_battery, gpu_offloaded) tuple the helper has been called with this process. The first time a tuple is seen, the helper logs at INFO via tracing! with the chosen thread count and the active clamps. Subsequent calls with the same tuple are silent. Without this guard the journal would accumulate one INFO per inference chunk.

Data flow / contract

• Pure read; no I/O beyond reading the env var (cheap) and probing power (cached 10 s).
• Idempotent within a tuple. Different tuples each get one INFO line.
• The env override is intentional: it lets users on unusual silicon (32-core Threadripper with a noisy thermal design, big.LITTLE asymmetric cores, etc) override the heuristic.

Tests

8 tests in crates/core/src/tuning.rs:106-232. The pattern is to use with_thread_env_lock (a static Mutex<()> to serialise env-var writes) plus power::with_override to drive the power probe.

• matches_existing_clamp_when_no_clamps_apply — bare call returns within [2, 8].
• env_var_bypasses_clamps — MAGNOTIA_INFERENCE_THREADS=10 returns 10 even with battery + GPU clamps active.
• workload_variants_distinct — sanity.
• battery_halves_thread_count — measures both states; battery should be ≤ AC.
• gpu_offload_clamps_llm_to_floor / gpu_offload_clamps_whisper_to_floor — GPU clamp.
• whisper_gpu_floor_is_at_least_llm_gpu_floor — architectural invariant.
• gpu_offload_off_does_not_clamp_below_battery_calc — GPU clamp does not undo the battery halve when GPU is off.
• logging_does_not_panic — smoke for the tracing path.

Watch-outs

• Two static Mutexes: SEEN and TEST_LOCK (in tests). Holding either across a panic poisons it. The helper uses if let Ok(mut seen) = log_seen().lock() rather than .expect() so a poisoned SEEN mutex degrades silently to "log every call" rather than panicking.
• MAGNOTIA_INFERENCE_THREADS is a process-global env-var. Per-call overrides need to wrap the call site in a temporary env-var, which is racy across threads. Tests serialise via THREAD_ENV_LOCK.
• Battery probe is a cross-process state. A user toggling the dock between calls sees the change after the 10-second power cache TTL. Acceptable for thread tuning; consumers wanting near-real-time should call force_clear_cache (test-only) or accept the 10 s lag.
• probe_gpu() returning None does not affect this helper. The gpu_offloaded flag is passed in by the caller (the LLM engine sets it from use_gpu && gpu_layers >= n_layer()). See the slice-4 LLM page for the call site.

See also

• Power-state probe
• Constants module (RAM thresholds)
• Slice 3 transcription engines
• Slice 4 LLM engine
• docs/superpowers/specs/2026-05-09-battery-gpu-aware-thread-tuning-design.md for the full design rationale.