jars
|
4cb954ece4
|
perf: route whisper + llm n_threads through physical-core helper
check / cargo check (macos-latest) (push) Has been cancelled
check / cargo check (ubuntu-22.04) (push) Has been cancelled
check / cargo check (windows-latest) (push) Has been cancelled
check / svelte build + lint (push) Has been cancelled
Both inference call sites previously called `num_cpus::get()` (logical
thread count). Established whisper.cpp / llama.cpp guidance is that
SMT siblings contend for shared FPU resources during heavy F16/F32
matmul, so going past physical core count anti-scales. Empirical
sweep on Whisper Tiny / 11s JFK clip / Ryzen 5 4650U (6c12t):
n_threads | xc_time | RTF | speedup_vs_1
----------|---------|--------|-------------
1 | 0.33s | 0.030 | 1.00x
2 | 0.33s | 0.030 | 1.00x
4 | 0.30s | 0.028 | 1.09x
6 | 0.32s | 0.029 | 1.04x
8 | 0.31s | 0.028 | 1.06x
12 | 0.32s | 0.029 | 1.03x
Tiny doesn't scale (work dominated by overhead) but the larger
Whisper variants and Qwen LLMs do. Sources: whisper.cpp #200, #1033,
#1252, #403; llama.cpp #3167, #572.
Changes:
- crates/core/src/constants.rs:
* MIN_INFERENCE_THREADS lowered 4 → 2 (research-derived floor)
* MAX_INFERENCE_THREADS = 8 added (research-derived ceiling)
* inference_thread_count() rewritten:
- reads MAGNOTIA_INFERENCE_THREADS env var (override)
- num_cpus::get_physical() with available_parallelism fallback
- clamped to [MIN_INFERENCE_THREADS, MAX_INFERENCE_THREADS]
- crates/core/Cargo.toml: + num_cpus = "1"
- crates/transcription/src/whisper_rs_backend.rs: call site uses helper.
- crates/llm/src/lib.rs: call site uses helper.
- crates/transcription/Cargo.toml: drop num_cpus from [features] +
[dependencies] (production no longer needs it). Move to
[dev-dependencies] for tests/thread_sweep.rs only.
- crates/llm/Cargo.toml: drop num_cpus = "1" (no longer used directly).
Per-machine maps (after this patch):
Ryzen 5 4650U 6c12t → 6
big-iron 12c24t → 8 (clamp; users can override)
cheap 2c2t laptop → 2
1c container/VM → 2
Adds crates/transcription/tests/thread_sweep.rs — env-gated like
jfk_bench, prints the table above against any model + WAV. Useful for
re-baselining on new hardware or when tuning the clamp values.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
2026-05-09 09:26:23 +01:00 |
|
jake
|
6588130e36
|
feat(kon): add core crate — types, traits, hardware, model registry, recommendation
- Value objects: ModelId, EngineName, Megabytes, AudioSamples, Segment, Transcript
- KonError enum with thiserror
- Constants centralised: audio pipeline, VAD, RAM thresholds, inference threading
- SpeechToText and TextProcessor provider traits with ProviderRegistry
- Unified model registry (Whisper tiny/base/small/medium + Parakeet CTC int8)
- Hardware detection: probe_ram, probe_cpu, probe_gpu (stub), probe_os
- Recommendation engine: score_model (pure function), rank_recommendations (sorted)
- 5 tests passing, clippy clean
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
2026-03-16 20:27:27 +00:00 |
|