perf: route whisper + llm n_threads through physical-core helper
Some checks failed
check / cargo check (macos-latest) (push) Has been cancelled
check / cargo check (ubuntu-22.04) (push) Has been cancelled
check / cargo check (windows-latest) (push) Has been cancelled
check / svelte build + lint (push) Has been cancelled

Both inference call sites previously called `num_cpus::get()` (logical
thread count). Established whisper.cpp / llama.cpp guidance is that
SMT siblings contend for shared FPU resources during heavy F16/F32
matmul, so going past physical core count anti-scales. Empirical
sweep on Whisper Tiny / 11s JFK clip / Ryzen 5 4650U (6c12t):

  n_threads | xc_time | RTF    | speedup_vs_1
  ----------|---------|--------|-------------
          1 |   0.33s |  0.030 |   1.00x
          2 |   0.33s |  0.030 |   1.00x
          4 |   0.30s |  0.028 |   1.09x
          6 |   0.32s |  0.029 |   1.04x
          8 |   0.31s |  0.028 |   1.06x
         12 |   0.32s |  0.029 |   1.03x

Tiny doesn't scale (work dominated by overhead) but the larger
Whisper variants and Qwen LLMs do. Sources: whisper.cpp #200, #1033,
#1252, #403; llama.cpp #3167, #572.

Changes:
- crates/core/src/constants.rs:
  * MIN_INFERENCE_THREADS lowered 4 → 2 (research-derived floor)
  * MAX_INFERENCE_THREADS = 8 added (research-derived ceiling)
  * inference_thread_count() rewritten:
      - reads MAGNOTIA_INFERENCE_THREADS env var (override)
      - num_cpus::get_physical() with available_parallelism fallback
      - clamped to [MIN_INFERENCE_THREADS, MAX_INFERENCE_THREADS]
- crates/core/Cargo.toml: + num_cpus = "1"
- crates/transcription/src/whisper_rs_backend.rs: call site uses helper.
- crates/llm/src/lib.rs: call site uses helper.
- crates/transcription/Cargo.toml: drop num_cpus from [features] +
  [dependencies] (production no longer needs it). Move to
  [dev-dependencies] for tests/thread_sweep.rs only.
- crates/llm/Cargo.toml: drop num_cpus = "1" (no longer used directly).

Per-machine maps (after this patch):
  Ryzen 5 4650U  6c12t → 6
  big-iron 12c24t       → 8 (clamp; users can override)
  cheap 2c2t laptop     → 2
  1c container/VM       → 2

Adds crates/transcription/tests/thread_sweep.rs — env-gated like
jfk_bench, prints the table above against any model + WAV. Useful for
re-baselining on new hardware or when tuning the clamp values.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jars
2026-05-09 09:26:23 +01:00
parent fdf27db0a1
commit 4cb954ece4
8 changed files with 134 additions and 16 deletions

View File

@@ -17,7 +17,7 @@ build = "build.rs"
# but skip the Vulkan backend. Build CPU-only with:
# cargo build -p magnotia-transcription --no-default-features --features whisper
default = ["whisper", "whisper-vulkan"]
whisper = ["dep:whisper-rs", "dep:num_cpus"]
whisper = ["dep:whisper-rs"]
whisper-vulkan = ["whisper-rs?/vulkan"]
[dependencies]
@@ -40,10 +40,6 @@ sha2 = "0.10"
# additive via the `whisper-vulkan` feature so non-GPU targets can drop it.
whisper-rs = { version = "0.16", default-features = false, optional = true }
# Direct whisper-rs backend (WhisperRsBackend): thread pool sizing.
# Gated alongside whisper-rs since no other code in this crate needs it.
num_cpus = { version = "1", optional = true }
# Typed error enum used by WhisperRsBackend + elsewhere. Kept
# unconditional because it is a derive-macro crate with negligible
# build cost.
@@ -56,3 +52,7 @@ tracing = "0.1"
# TcpListener fixture for the download resume tests (mirrors magnotia-llm).
tokio = { version = "1", features = ["rt", "sync", "net", "io-util", "macros"] }
tempfile = "3"
# Test-only — used by tests/thread_sweep.rs to label physical vs logical
# core counts in the scaling table. Production code uses the
# `magnotia_core::constants::inference_thread_count` helper instead.
num_cpus = "1"