4cb954ece4
Both inference call sites previously called `num_cpus::get()` (logical
thread count). Established whisper.cpp / llama.cpp guidance is that
SMT siblings contend for shared FPU resources during heavy F16/F32
matmul, so going past physical core count anti-scales. Empirical
sweep on Whisper Tiny / 11s JFK clip / Ryzen 5 4650U (6c12t):
n_threads | xc_time | RTF | speedup_vs_1
----------|---------|--------|-------------
1 | 0.33s | 0.030 | 1.00x
2 | 0.33s | 0.030 | 1.00x
4 | 0.30s | 0.028 | 1.09x
6 | 0.32s | 0.029 | 1.04x
8 | 0.31s | 0.028 | 1.06x
12 | 0.32s | 0.029 | 1.03x
Tiny doesn't scale (work dominated by overhead) but the larger
Whisper variants and Qwen LLMs do. Sources: whisper.cpp #200, #1033,
#1252, #403; llama.cpp #3167, #572.
Changes:
- crates/core/src/constants.rs:
* MIN_INFERENCE_THREADS lowered 4 → 2 (research-derived floor)
* MAX_INFERENCE_THREADS = 8 added (research-derived ceiling)
* inference_thread_count() rewritten:
- reads MAGNOTIA_INFERENCE_THREADS env var (override)
- num_cpus::get_physical() with available_parallelism fallback
- clamped to [MIN_INFERENCE_THREADS, MAX_INFERENCE_THREADS]
- crates/core/Cargo.toml: + num_cpus = "1"
- crates/transcription/src/whisper_rs_backend.rs: call site uses helper.
- crates/llm/src/lib.rs: call site uses helper.
- crates/transcription/Cargo.toml: drop num_cpus from [features] +
[dependencies] (production no longer needs it). Move to
[dev-dependencies] for tests/thread_sweep.rs only.
- crates/llm/Cargo.toml: drop num_cpus = "1" (no longer used directly).
Per-machine maps (after this patch):
Ryzen 5 4650U 6c12t → 6
big-iron 12c24t → 8 (clamp; users can override)
cheap 2c2t laptop → 2
1c container/VM → 2
Adds crates/transcription/tests/thread_sweep.rs — env-gated like
jfk_bench, prints the table above against any model + WAV. Useful for
re-baselining on new hardware or when tuning the clamp values.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
59 lines
2.2 KiB
TOML
59 lines
2.2 KiB
TOML
[package]
|
|
name = "magnotia-transcription"
|
|
version = "0.1.0"
|
|
edition = "2021"
|
|
description = "Speech-to-text engine wrappers, model management, and inference concurrency for Magnotia"
|
|
build = "build.rs"
|
|
|
|
[features]
|
|
# Whisper backend (direct whisper-rs). Default on — gating it exists so
|
|
# a future Windows non-AVX2 build, or a cloud-only ASR configuration,
|
|
# can drop whisper-rs-sys entirely per brief item #13. Disabling this
|
|
# feature also drops the WhisperRsBackend module and the load_whisper
|
|
# entry point.
|
|
#
|
|
# `whisper-vulkan` is a separate feature so a non-Vulkan target (Android
|
|
# without GPU drivers, a CPU-only Windows build) can pull in whisper-rs
|
|
# but skip the Vulkan backend. Build CPU-only with:
|
|
# cargo build -p magnotia-transcription --no-default-features --features whisper
|
|
default = ["whisper", "whisper-vulkan"]
|
|
whisper = ["dep:whisper-rs"]
|
|
whisper-vulkan = ["whisper-rs?/vulkan"]
|
|
|
|
[dependencies]
|
|
magnotia-core = { path = "../core" }
|
|
|
|
# Parakeet via ONNX. Whisper is handled directly via whisper-rs below.
|
|
transcribe-rs = { version = "0.3", default-features = false, features = ["onnx"] }
|
|
|
|
# Async runtime for spawn_blocking
|
|
tokio = { version = "1", features = ["rt", "sync"] }
|
|
|
|
# Model downloads
|
|
reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "stream"] }
|
|
futures-util = "0.3"
|
|
|
|
# Download integrity verification
|
|
sha2 = "0.10"
|
|
|
|
# Gated behind the `whisper` feature (see [features] above). Vulkan is
|
|
# additive via the `whisper-vulkan` feature so non-GPU targets can drop it.
|
|
whisper-rs = { version = "0.16", default-features = false, optional = true }
|
|
|
|
# Typed error enum used by WhisperRsBackend + elsewhere. Kept
|
|
# unconditional because it is a derive-macro crate with negligible
|
|
# build cost.
|
|
thiserror = "2"
|
|
|
|
# Structured logging at backend boundaries (observability for initial_prompt flow).
|
|
tracing = "0.1"
|
|
|
|
[dev-dependencies]
|
|
# TcpListener fixture for the download resume tests (mirrors magnotia-llm).
|
|
tokio = { version = "1", features = ["rt", "sync", "net", "io-util", "macros"] }
|
|
tempfile = "3"
|
|
# Test-only — used by tests/thread_sweep.rs to label physical vs logical
|
|
# core counts in the scaling table. Production code uses the
|
|
# `magnotia_core::constants::inference_thread_count` helper instead.
|
|
num_cpus = "1"
|