docs: architecture map (initial 5-slice generation, 105 pages)
Five-slice navigable map of the entire codebase under
docs/architecture-map/. Each slice is a self-contained
breadcrumbed sub-tree:
01-frontend (16) Svelte/SvelteKit UI
02-tauri-runtime (26) src-tauri commands + lifecycle
03-audio-transcription (16) audio + transcription crates
04-llm-formatting-mcp (19) llm, ai-formatting, mcp, cloud
05-core-storage-hotkey-build core, storage, hotkey, workspace,
(26) CI, dev glue
Plus master README.md and data-flow-end-to-end.md tracing
audio bytes from microphone to FTS5 search to MCP read.
Generated by 5 parallel subagents on 2026/05/09 against
HEAD 3c47000. Each page has YAML frontmatter, file:line code
refs, sibling cross-links, plain-English summaries.
Aggregated debt surfaced (full lists in master README):
RB-08 macOS power assertion, schema head drift v14 vs v15,
VAD blocked on ort version conflict, streaming primitives
not wired into live.rs, no prompt versioning, MCP has no
auth, cloud-providers in-memory keystore, SettingsPage
2 484 LOC, commands/live.rs 1 737 LOC, dual theme system,
brand rename to Lumenote pending across the codebase.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jars
@@ -0,0 +1,95 @@
|
||||
---
|
||||
name: Core inference thread tuning
|
||||
type: architecture-map-page
|
||||
slice: 05-core-storage-hotkey-build
|
||||
last_verified: 2026/05/09
|
||||
---
|
||||
|
||||
# Core inference thread tuning
|
||||
|
||||
> **Where you are:** [Architecture map](../README.md) → [Core, Storage, Hotkey, Build](README.md) → Core inference thread tuning
|
||||
|
||||
**Plain English summary.** A single helper, `inference_thread_count`, that answers "how many CPU threads should we give whisper.cpp / llama.cpp on this user's machine right now?" Combines the physical-core budget, battery awareness, and GPU-offload awareness into one number. Logs the chosen value once per (workload, battery, GPU-offload) tuple so the same configuration does not flood the journal.
|
||||
|
||||
## At a glance
|
||||
|
||||
- File: `crates/core/src/tuning.rs` (233 LOC, 127 of which are tests).
|
||||
- External deps: `num_cpus 1`, `tracing 0.1`, plus the in-crate `power` module.
|
||||
- Public surface: `MIN_INFERENCE_THREADS`, `MAX_INFERENCE_THREADS`, `Workload` enum, `inference_thread_count`.
|
||||
- Consumers: slice 3 (Whisper engine, Parakeet engine — `Workload::Whisper`); slice 4 (LLM engine — `Workload::Llm`).
|
||||
|
||||
## What's in here
|
||||
|
||||
### Constants — `crates/core/src/tuning.rs:11, 20`
|
||||
|
||||
```rust
|
||||
pub const MIN_INFERENCE_THREADS: usize = 2;
|
||||
pub const MAX_INFERENCE_THREADS: usize = 8;
|
||||
```
|
||||
|
||||
Single-threaded inference is measurably worse than two on every multi-core part. Past 8 threads, `whisper.cpp` and `llama.cpp` scaling flattens because SMT siblings contend for the FPU during F16 / F32 matmul. 8 is a conservative ceiling that leaves <5% on the table for big-iron desktops while keeping consumer 6c/12t laptops out of contention territory.
|
||||
|
||||
### `Workload` enum — `crates/core/src/tuning.rs:23`
|
||||
|
||||
```rust
|
||||
pub enum Workload { Llm, Whisper }
|
||||
```
|
||||
|
||||
The two workloads have different floors when GPU offload is on.
|
||||
|
||||
### Internal floors — `crates/core/src/tuning.rs:33-34`
|
||||
|
||||
```rust
|
||||
const GPU_FLOOR_LLM: usize = 2;
|
||||
const GPU_FLOOR_WHISPER: usize = 4;
|
||||
```
|
||||
|
||||
Whisper retains CPU work even with full Vulkan offload (mel spectrogram, decoder bookkeeping, beam search). LLM (llama-style transformer) drops to near-zero CPU work when fully offloaded. Architectural invariant tested at `tuning.rs:200`: `GPU_FLOOR_WHISPER >= GPU_FLOOR_LLM`.
|
||||
|
||||
### `inference_thread_count(workload, gpu_offloaded) -> usize` — `crates/core/src/tuning.rs:53`
|
||||
|
||||
Resolution order:
|
||||
|
||||
1. **`MAGNOTIA_INFERENCE_THREADS=N`** — absolute bypass, returns `N` without any clamps.
|
||||
2. **Base** = `num_cpus::get_physical()`. Falls back to `std::thread::available_parallelism()`, then to `MIN_INFERENCE_THREADS = 2`.
|
||||
3. **On battery** → `base /= 2`. Power state is read from `magnotia_core::power::probe_power_state()`. The 10-second cache there means the call is cheap; see [`core-power.md`](core-power.md).
|
||||
4. **GPU offloaded** → clamp to `GPU_FLOOR_<workload>`. Whisper floor is 4, LLM floor is 2.
|
||||
5. **Final clamp** to `[MIN_INFERENCE_THREADS, MAX_INFERENCE_THREADS]`.
|
||||
|
||||
### One-shot logging — `crates/core/src/tuning.rs:39, 87-101`
|
||||
|
||||
A `static SEEN: OnceLock<Mutex<HashSet<(Workload, bool, bool)>>>` records every distinct `(workload, on_battery, gpu_offloaded)` tuple the helper has been called with this process. The first time a tuple is seen, the helper logs at `INFO` via `tracing!` with the chosen thread count and the active clamps. Subsequent calls with the same tuple are silent. Without this guard the journal would accumulate one INFO per inference chunk.
|
||||
|
||||
## Data flow / contract
|
||||
|
||||
- Pure read; no I/O beyond reading the env var (cheap) and probing power (cached 10 s).
|
||||
- Idempotent within a tuple. Different tuples each get one INFO line.
|
||||
- The env override is intentional: it lets users on unusual silicon (32-core Threadripper with a noisy thermal design, big.LITTLE asymmetric cores, etc) override the heuristic.
|
||||
|
||||
## Tests
|
||||
|
||||
8 tests in `crates/core/src/tuning.rs:106-232`. The pattern is to use `with_thread_env_lock` (a static `Mutex<()>` to serialise env-var writes) plus `power::with_override` to drive the power probe.
|
||||
|
||||
- `matches_existing_clamp_when_no_clamps_apply` — bare call returns within `[2, 8]`.
|
||||
- `env_var_bypasses_clamps` — `MAGNOTIA_INFERENCE_THREADS=10` returns 10 even with battery + GPU clamps active.
|
||||
- `workload_variants_distinct` — sanity.
|
||||
- `battery_halves_thread_count` — measures both states; battery should be ≤ AC.
|
||||
- `gpu_offload_clamps_llm_to_floor` / `gpu_offload_clamps_whisper_to_floor` — GPU clamp.
|
||||
- `whisper_gpu_floor_is_at_least_llm_gpu_floor` — architectural invariant.
|
||||
- `gpu_offload_off_does_not_clamp_below_battery_calc` — GPU clamp does not undo the battery halve when GPU is off.
|
||||
- `logging_does_not_panic` — smoke for the tracing path.
|
||||
|
||||
## Watch-outs
|
||||
|
||||
- **Two static `Mutex`es: `SEEN` and `TEST_LOCK` (in tests).** Holding either across a panic poisons it. The helper uses `if let Ok(mut seen) = log_seen().lock()` rather than `.expect()` so a poisoned `SEEN` mutex degrades silently to "log every call" rather than panicking.
|
||||
- **`MAGNOTIA_INFERENCE_THREADS` is a process-global env-var.** Per-call overrides need to wrap the call site in a temporary env-var, which is racy across threads. Tests serialise via `THREAD_ENV_LOCK`.
|
||||
- **Battery probe is a cross-process state.** A user toggling the dock between calls sees the change after the 10-second power cache TTL. Acceptable for thread tuning; consumers wanting near-real-time should call `force_clear_cache` (test-only) or accept the 10 s lag.
|
||||
- **`probe_gpu()` returning `None` does not affect this helper.** The `gpu_offloaded` flag is passed in by the caller (the LLM engine sets it from `use_gpu && gpu_layers >= n_layer()`). See the slice-4 LLM page for the call site.
|
||||
|
||||
## See also
|
||||
|
||||
- [Power-state probe](core-power.md)
|
||||
- [Constants module](core-constants.md) (RAM thresholds)
|
||||
- [Slice 3 transcription engines](../03-audio-transcription/README.md)
|
||||
- [Slice 4 LLM engine](../04-llm-formatting-mcp/README.md)
|
||||
- `docs/superpowers/specs/2026-05-09-battery-gpu-aware-thread-tuning-design.md` for the full design rationale.
|
||||
Reference in New Issue
Block a user