jars
7eb69e6251
refactor(core): remove old inference_thread_count facade
...
The MIN_INFERENCE_THREADS, MAX_INFERENCE_THREADS, and
inference_thread_count() defined in constants.rs were superseded by
the equivalents in tuning.rs (Task 2.1) and are no longer called by
any production code. Both call sites (whisper backend Task 4.1, LLM
generate Task 5.1) have migrated.
Constants.rs now holds only audio/mel/RAM/VAD/download constants;
inference-tuning concerns live in tuning.rs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-05-09 12:41:23 +01:00
jars
4cb954ece4
perf: route whisper + llm n_threads through physical-core helper
...
check / cargo check (macos-latest) (push) Has been cancelled
check / cargo check (ubuntu-22.04) (push) Has been cancelled
check / cargo check (windows-latest) (push) Has been cancelled
check / svelte build + lint (push) Has been cancelled
Both inference call sites previously called `num_cpus::get()` (logical
thread count). Established whisper.cpp / llama.cpp guidance is that
SMT siblings contend for shared FPU resources during heavy F16/F32
matmul, so going past physical core count anti-scales. Empirical
sweep on Whisper Tiny / 11s JFK clip / Ryzen 5 4650U (6c12t):
n_threads | xc_time | RTF | speedup_vs_1
----------|---------|--------|-------------
1 | 0.33s | 0.030 | 1.00x
2 | 0.33s | 0.030 | 1.00x
4 | 0.30s | 0.028 | 1.09x
6 | 0.32s | 0.029 | 1.04x
8 | 0.31s | 0.028 | 1.06x
12 | 0.32s | 0.029 | 1.03x
Tiny doesn't scale (work dominated by overhead) but the larger
Whisper variants and Qwen LLMs do. Sources: whisper.cpp #200 , #1033 ,
#1252 , #403 ; llama.cpp #3167 , #572 .
Changes:
- crates/core/src/constants.rs:
* MIN_INFERENCE_THREADS lowered 4 → 2 (research-derived floor)
* MAX_INFERENCE_THREADS = 8 added (research-derived ceiling)
* inference_thread_count() rewritten:
- reads MAGNOTIA_INFERENCE_THREADS env var (override)
- num_cpus::get_physical() with available_parallelism fallback
- clamped to [MIN_INFERENCE_THREADS, MAX_INFERENCE_THREADS]
- crates/core/Cargo.toml: + num_cpus = "1"
- crates/transcription/src/whisper_rs_backend.rs: call site uses helper.
- crates/llm/src/lib.rs: call site uses helper.
- crates/transcription/Cargo.toml: drop num_cpus from [features] +
[dependencies] (production no longer needs it). Move to
[dev-dependencies] for tests/thread_sweep.rs only.
- crates/llm/Cargo.toml: drop num_cpus = "1" (no longer used directly).
Per-machine maps (after this patch):
Ryzen 5 4650U 6c12t → 6
big-iron 12c24t → 8 (clamp; users can override)
cheap 2c2t laptop → 2
1c container/VM → 2
Adds crates/transcription/tests/thread_sweep.rs — env-gated like
jfk_bench, prints the table above against any model + WAV. Useful for
re-baselining on new hardware or when tuning the clamp values.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-05-09 09:26:23 +01:00
jake
6588130e36
feat(kon): add core crate — types, traits, hardware, model registry, recommendation
...
- Value objects: ModelId, EngineName, Megabytes, AudioSamples, Segment, Transcript
- KonError enum with thiserror
- Constants centralised: audio pipeline, VAD, RAM thresholds, inference threading
- SpeechToText and TextProcessor provider traits with ProviderRegistry
- Unified model registry (Whisper tiny/base/small/medium + Parakeet CTC int8)
- Hardware detection: probe_ram, probe_cpu, probe_gpu (stub), probe_os
- Recommendation engine: score_model (pure function), rank_recommendations (sorted)
- 5 tests passing, clippy clean
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
2026-03-16 20:27:27 +00:00
