---
name: Core inference thread tuning
type: architecture-map-page
slice: 05-core-storage-hotkey-build
last_verified: 2026/05/09
---

# Core inference thread tuning

> **Where you are:** [Architecture map](../README.md) → [Core, Storage, Hotkey, Build](README.md) → Core inference thread tuning

**Plain English summary.** A single helper, `inference_thread_count`, that answers "how many CPU threads should we give whisper.cpp / llama.cpp on this user's machine right now?" Combines the physical-core budget, battery awareness, and GPU-offload awareness into one number. Logs the chosen value once per (workload, battery, GPU-offload) tuple so the same configuration does not flood the journal.

## At a glance

- File: `crates/core/src/tuning.rs` (233 LOC, 127 of which are tests).
- External deps: `num_cpus 1`, `tracing 0.1`, plus the in-crate `power` module.
- Public surface: `MIN_INFERENCE_THREADS`, `MAX_INFERENCE_THREADS`, `Workload` enum, `inference_thread_count`.
- Consumers: slice 3 (Whisper engine, Parakeet engine — `Workload::Whisper`); slice 4 (LLM engine — `Workload::Llm`).

## What's in here

### Constants — `crates/core/src/tuning.rs:11, 20`

```rust
pub const MIN_INFERENCE_THREADS: usize = 2;
pub const MAX_INFERENCE_THREADS: usize = 8;
```

Single-threaded inference is measurably worse than two on every multi-core part. Past 8 threads, `whisper.cpp` and `llama.cpp` scaling flattens because SMT siblings contend for the FPU during F16 / F32 matmul. 8 is a conservative ceiling that leaves <5% on the table for big-iron desktops while keeping consumer 6c/12t laptops out of contention territory.

### `Workload` enum — `crates/core/src/tuning.rs:23`

```rust
pub enum Workload { Llm, Whisper }
```

The two workloads have different floors when GPU offload is on.

### Internal floors — `crates/core/src/tuning.rs:33-34`

```rust
const GPU_FLOOR_LLM: usize = 2;
const GPU_FLOOR_WHISPER: usize = 4;
```

Whisper retains CPU work even with full Vulkan offload (mel spectrogram, decoder bookkeeping, beam search). LLM (llama-style transformer) drops to near-zero CPU work when fully offloaded. Architectural invariant tested at `tuning.rs:200`: `GPU_FLOOR_WHISPER >= GPU_FLOOR_LLM`.

### `inference_thread_count(workload, gpu_offloaded) -> usize` — `crates/core/src/tuning.rs:53`

Resolution order:

1. **`MAGNOTIA_INFERENCE_THREADS=N`** — absolute bypass, returns `N` without any clamps.
2. **Base** = `num_cpus::get_physical()`. Falls back to `std::thread::available_parallelism()`, then to `MIN_INFERENCE_THREADS = 2`.
3. **On battery** → `base /= 2`. Power state is read from `lumotia_core::power::probe_power_state()`. The 10-second cache there means the call is cheap; see [`core-power.md`](core-power.md).
4. **GPU offloaded** → clamp to `GPU_FLOOR_<workload>`. Whisper floor is 4, LLM floor is 2.
5. **Final clamp** to `[MIN_INFERENCE_THREADS, MAX_INFERENCE_THREADS]`.

### One-shot logging — `crates/core/src/tuning.rs:39, 87-101`

A `static SEEN: OnceLock<Mutex<HashSet<(Workload, bool, bool)>>>` records every distinct `(workload, on_battery, gpu_offloaded)` tuple the helper has been called with this process. The first time a tuple is seen, the helper logs at `INFO` via `tracing!` with the chosen thread count and the active clamps. Subsequent calls with the same tuple are silent. Without this guard the journal would accumulate one INFO per inference chunk.

## Data flow / contract

- Pure read; no I/O beyond reading the env var (cheap) and probing power (cached 10 s).
- Idempotent within a tuple. Different tuples each get one INFO line.
- The env override is intentional: it lets users on unusual silicon (32-core Threadripper with a noisy thermal design, big.LITTLE asymmetric cores, etc) override the heuristic.

## Tests

8 tests in `crates/core/src/tuning.rs:106-232`. The pattern is to use `with_thread_env_lock` (a static `Mutex<()>` to serialise env-var writes) plus `power::with_override` to drive the power probe.

- `matches_existing_clamp_when_no_clamps_apply` — bare call returns within `[2, 8]`.
- `env_var_bypasses_clamps` — `MAGNOTIA_INFERENCE_THREADS=10` returns 10 even with battery + GPU clamps active.
- `workload_variants_distinct` — sanity.
- `battery_halves_thread_count` — measures both states; battery should be ≤ AC.
- `gpu_offload_clamps_llm_to_floor` / `gpu_offload_clamps_whisper_to_floor` — GPU clamp.
- `whisper_gpu_floor_is_at_least_llm_gpu_floor` — architectural invariant.
- `gpu_offload_off_does_not_clamp_below_battery_calc` — GPU clamp does not undo the battery halve when GPU is off.
- `logging_does_not_panic` — smoke for the tracing path.

## Watch-outs

- **Two static `Mutex`es: `SEEN` and `TEST_LOCK` (in tests).** Holding either across a panic poisons it. The helper uses `if let Ok(mut seen) = log_seen().lock()` rather than `.expect()` so a poisoned `SEEN` mutex degrades silently to "log every call" rather than panicking.
- **`MAGNOTIA_INFERENCE_THREADS` is a process-global env-var.** Per-call overrides need to wrap the call site in a temporary env-var, which is racy across threads. Tests serialise via `THREAD_ENV_LOCK`.
- **Battery probe is a cross-process state.** A user toggling the dock between calls sees the change after the 10-second power cache TTL. Acceptable for thread tuning; consumers wanting near-real-time should call `force_clear_cache` (test-only) or accept the 10 s lag.
- **`probe_gpu()` returning `None` does not affect this helper.** The `gpu_offloaded` flag is passed in by the caller (the LLM engine sets it from `use_gpu && gpu_layers >= n_layer()`). See the slice-4 LLM page for the call site.

## See also

- [Power-state probe](core-power.md)
- [Constants module](core-constants.md) (RAM thresholds)
- [Slice 3 transcription engines](../03-audio-transcription/README.md)
- [Slice 4 LLM engine](../04-llm-formatting-mcp/README.md)
- `docs/superpowers/specs/2026-05-09-battery-gpu-aware-thread-tuning-design.md` for the full design rationale.