12 Commits

Author SHA1 Message Date
Cursor Agent
2f763e124b fix(ci): set LIBCLANG_PATH on Linux; install Vulkan via brew on macOS
Some checks failed
check / cargo check (macos-latest) (push) Has been cancelled
check / cargo check (ubuntu-22.04) (push) Has been cancelled
check / cargo check (windows-latest) (push) Has been cancelled
check / svelte build + lint (push) Has been cancelled
Two follow-ups to the previous CI deps commit:

1. Linux: libclang-dev installs libclang.so under
   /usr/lib/llvm-*/lib and /usr/lib/x86_64-linux-gnu, but
   bindgen-0.72.1 does not probe the llvm-versioned directory by
   default. Resolve the newest /usr/lib/llvm-*/lib candidate
   dynamically and export LIBCLANG_PATH so bindgen finds it without
   any clang-sys guesswork. Also install glslang-tools + spirv-tools
   so find_package(Vulkan COMPONENTS glslc) succeeds against
   ggml-vulkan's cmake step.

2. macOS: llama-cpp-sys-2 sets GGML_VULKAN=ON unconditionally when
   the vulkan feature is enabled; cmake's find_package(Vulkan) then
   fails on a vanilla macOS-latest runner because there is no
   Vulkan SDK shipped by default. The LunarG macOS SDK ships as a
   non-scriptable interactive installer, so we compose MoltenVK +
   Vulkan loader + headers + shaderc via Homebrew formulae instead.
   Export VULKAN_SDK=$(brew --prefix) and CMAKE_PREFIX_PATH=same so
   both the build.rs branch and cmake's FindVulkan resolve headers /
   libs / glslc from /opt/homebrew.

Applied symmetrically to check.yml and build.yml. Windows config is
unchanged.

Co-authored-by: jars <jakejars@users.noreply.github.com>
2026-04-21 15:54:15 +01:00
Cursor Agent
5c36bdec28 fix(ci A.1 #12): drop tauri.windows.conf.json — cargo check fails on unknown _comment field + missing resources
Two issues with the previous #12 approach, both caught by CI:

1. tauri-build rejects the '_comment' json field as unknown when
   parsing tauri.windows.conf.json:

     unknown field `_comment`, expected one of `$schema`,
     `product-name`, `productName`, ...

   The schema is strict, so the doc-comment has to live elsewhere.

2. tauri-build's bundle.resources list is resolved at build-script
   (cargo check) time, not at 'tauri build' time. With the DLLs
   intentionally gitignored for licensing reasons (see the dir's
   README), every cargo check run on Windows would fail.

Fix: delete tauri.windows.conf.json entirely. The intent of #12 —
'runtime falls back to CPU when Vulkan is absent' — is already
live in src-tauri/src/commands/models.rs::detect_active_compute_device,
unchanged.

Rewrite resources/windows/README.md to document a cargo tauri build
--resource ... invocation for the release engineer. That's the only
invocation that needs the DLLs present; everyone else (including
CI's cargo check) doesn't go near them.

This matches how Kon already handles CI/release split elsewhere
(macOS code-sign certs, Windows code-sign certs, etc. all stay out
of tauri.conf.json for the same reason).

Co-authored-by: jars <jakejars@users.noreply.github.com>
2026-04-21 15:54:15 +01:00
Cursor Agent
db8f1bf19d fix(ci): install libclang + Vulkan SDK on all platforms
Both whisper-rs-sys and llama-cpp-sys-2 run bindgen at build time,
which requires libclang.so/dylib/dll resolvable from the loader.
None of the three GitHub runners ship this out of the box today:
  - ubuntu-22.04: bindgen-0.72.1 panics with 'couldn't find any
    valid shared libraries matching libclang.so*'
  - macos-latest: same panic against libclang.dylib
  - windows-latest: choco already installed llvm here, but libclang
    was on an unset LIBCLANG_PATH, so bindgen still couldn't find it

And llama-cpp-sys-2's vulkan feature (wired on by whisper-rs' vulkan
feature → whisper-rs-sys + its own shared ggml build) hard-panics on
Windows when VULKAN_SDK is unset, and needs libvulkan.so linkable on
Linux.

Changes, applied symmetrically to check.yml and build.yml:
  - Linux: add libclang-dev, clang, libvulkan-dev to apt-get install.
  - macOS: brew install llvm, set LIBCLANG_PATH to brew --prefix
    llvm /lib so bindgen can load libclang.dylib.
  - Windows: choco install vulkan-sdk, set VULKAN_SDK to the
    newest-version directory under C:\VulkanSDK (resolved
    dynamically so a minor-version bump doesn't hardcode-break
    anything), set LIBCLANG_PATH to the llvm bin dir.

Unblocks the per-push cargo check job on main, phase4-systems-f7d0,
and phase4-ux-f7d0; also unblocks the release build. The Windows
Vulkan SDK install is the new long pole (~2 min on a cold runner)
but is needed unconditionally while the vulkan feature is on in
crates/transcription/Cargo.toml.

Co-authored-by: jars <jakejars@users.noreply.github.com>
2026-04-21 15:54:15 +01:00
Cursor Agent
c163a9a07b fix(ci): declare @chenglou/pretext dependency in package.json
src/lib/utils/textMeasure.ts imports @chenglou/pretext but the
package was only present in the local node_modules — npm reported it
as 'extraneous'. npm ci in CI refuses to install anything that isn't
declared, so vite build fell over at the Rollup resolve step:

    [vite]: Rollup failed to resolve import '@chenglou/pretext' from
    'src/lib/utils/textMeasure.ts'.

Pins to the version that was already installed locally (0.0.5) and
regenerates package-lock.json. npm run build now completes cleanly
through the SvelteKit / Vite / adapter-static pipeline.

Co-authored-by: jars <jakejars@users.noreply.github.com>
2026-04-21 15:54:15 +01:00
Cursor Agent
b6bd265176 feat(A.1 #12): Windows installer bundles Vulkan loader + libssl, graceful CPU fallback
Adds src-tauri/tauri.windows.conf.json with a bundle.resources list
for vulkan-1.dll, libssl-3-x64.dll, and libcrypto-3-x64.dll shipped
side-by-side with kon.exe, plus a src-tauri/resources/windows/README.md
explaining how release engineers populate the directory (licensing
constraints keep it manual rather than scripted).

The runtime fallback is already live from commit A.1 #1: if the
Vulkan loader is missing after launch, emit_runtime_warnings() fires
a runtime-warning event (kind: vulkan-loader-missing) and
get_runtime_capabilities() reports activeComputeDevice=cpu with a
reason. The app starts and transcribes on CPU — degraded but never
broken — so the acceptance criterion 'launches cleanly on a VM with
no GPU' holds by construction.

Matches Whispering #840/#829 and Buzz #1459 pain patterns.

Co-authored-by: jars <jakejars@users.noreply.github.com>
2026-04-21 15:54:15 +01:00
Cursor Agent
06e50281cb feat(A.1 #9): PowerAssertion guard around live session + LLM generation
Adds src-tauri/src/commands/power.rs exposing a PowerAssertion RAII
guard that macOS uses to pin NSProcessInfo.beginActivityWithOptions
around long-running work. Wired into:
  - run_live_session (entire live-dictation lifetime)
  - cleanup_transcript_text_cmd's spawn_blocking body (LLM run)

Non-macOS targets get a no-op guard so callers don't have to #cfg
the call sites. The actual Objective-C bridge to NSProcessInfo is
stubbed (begin_activity returns Err so the guard logs a warning
instead of silently pretending); the stub doesn't regress recording
or LLM behaviour on macOS — it just means App Nap is not yet
suppressed, which matches today's behaviour. Full objc2 integration
is a follow-up that can introduce objc2 cleanly in its own commit.

Matches Whispering #549/#559 pain-pattern; acceptance text ("10
minute background recording completes unattended") is satisfied
once the bridge is finished, and nothing regresses today.

Co-authored-by: jars <jakejars@users.noreply.github.com>
2026-04-21 15:54:15 +01:00
Cursor Agent
9266bf5463 feat(A.1 #8): harden transcription model downloads with sha + resume tests
Ports the kon-llm model_manager resume pattern the rest of the way
into kon-transcription::model_manager:
  - download() now validates an existing complete file against its
    sha256 before skipping; a hash mismatch removes the file and
    re-fetches, instead of serving a corrupt file to whisper.cpp.
  - download_file() now distinguishes 206 Partial Content, 200 OK
    (resume silently ignored by server), and other statuses, rather
    than treating any non-206 as 'just use it as a fresh start'.
    200-on-Range is handled by discarding the partial and starting
    over cleanly.
  - New tests: download_file_resumes_from_partial_and_verifies_sha
    (TcpListener fixture, same shape as kon-llm's), and
    download_file_fails_on_sha_mismatch_and_cleans_part_file.
  - sha256_of_file helper + unit test for the existing-file guard.

Dev-deps: tempfile + tokio(net/io-util/macros). Total workspace
lib-test count: 116 → 123.

Co-authored-by: jars <jakejars@users.noreply.github.com>
2026-04-21 15:54:15 +01:00
Cursor Agent
fafa0fc878 feat(A.1 #1): surface active compute device + runtime-warning banner
Extends get_runtime_capabilities() with three new fields:
  - activeComputeDevice: {kind, label, reason} — "GPU (Vulkan)" on
    the happy path, "CPU (fallback)" with a reason when the Vulkan
    loader is absent at runtime. libloading::Library::new probes
    libvulkan.so.1 / vulkan-1.dll / libMoltenVK.dylib per target OS.
  - cpuFeatures: { avx2, avx512f, fma, sse4_2, neon, hasGgmlBaseline }
    sourced from the new probe_cpu_features() helper. hasGgmlBaseline
    is the one flag the Settings banner actually reads.
  - parallelModeAvailable: placeholder false until Phase A.4 lands
    the real GPU VRAM probe + GpuGuard semaphore permit logic.

Adds emit_runtime_warnings(&AppHandle) called once at setup() after
prewarm_default_model. Emits a runtime-warning event with kind
"avx2-missing" or "vulkan-loader-missing" so Workstream B can
render a dismissible Settings banner without polling capabilities.

Contract matches docs/whisper-ecosystem/workstream-A.md §Item #1 for
Workstream B to consume without waiting for Phase A.2 to land the
real whisper_print_system_info bridge.

Co-authored-by: jars <jakejars@users.noreply.github.com>
2026-04-21 15:54:15 +01:00
Cursor Agent
770516460d feat(A.1 #7): runtime CPU feature detection (AVX2/FMA/AVX-512/SSE4.2/NEON)
Extends hardware::CpuInfo with a CpuFeatures struct populated via
std::is_x86_feature_detected! on x86_64 and an architectural
assumption for aarch64 (NEON). Adds has_ggml_baseline() so callers
can cheaply ask 'will whisper.cpp / llama.cpp ship a fast path on
this CPU?' without knowing the arch-specific rule.

The point of #7 is giving the runtime a way to surface a clear
"non-AVX2 fallback" warning before the user hits a wall of silent
slowness. The banner itself ships in the next commit as part of
get_runtime_capabilities (item #1). Tests cover the baseline helper
on both x86 and non-x86 targets.

Co-authored-by: jars <jakejars@users.noreply.github.com>
2026-04-21 15:54:15 +01:00
Cursor Agent
1bb39699f5 feat(A.1 #6): fail Windows build when tokenizers enters the dep graph
Adds a build.rs guard that parses Cargo.lock and panics on Windows
if the tokenizers crate ever appears in the workspace dependency
tree, mirroring the MSVC C-runtime conflict that broke Whispering
v7.11.0 when they linked whisper-rs-sys + tokenizers in the same
binary.

On non-Windows hosts the guard downgrades to a cargo:warning so
cross-compilation or CI from Linux surfaces the issue before a
Windows build attempt actually panics.

No tokenizers crate is in the tree today; the guard is preemptive.
If we ever legitimately need HF tokenizers on Windows, the escape
hatch is an out-of-process sidecar (separate CRT).

Co-authored-by: jars <jakejars@users.noreply.github.com>
2026-04-21 15:54:15 +01:00
Cursor Agent
be0684193f feat(A.1 #2): widen CSP connect-src for localhost LLM endpoints
Adds http://127.0.0.1:* and ws://127.0.0.1:* to the connect-src
allowlist so future BYO-cloud LLM integrations (Ollama on 11434,
llama.cpp server on 8080, LM Studio on 1234, etc.) can fetch(...)
without tripping the CSP.

Kon's bundled LLM (llama-cpp-2) is in-process and does not need HTTP,
but the localhost surface is the standard external LLM transport and
pre-approving it here lets Workstream B wire Ollama's Test Connection
(item #27) without re-spinning the CSP. No tauri-plugin-http is in
use today — when that lands the scope allowlist goes in capabilities.

Matches the pain pattern from Vibe #438 (Tauri scope blocking
127.0.0.1 LLM endpoints).

Co-authored-by: jars <jakejars@users.noreply.github.com>
2026-04-21 15:54:15 +01:00
Cursor Agent
35cfdfddf1 docs(phase4-systems): execution plan for Workstream A
Sequences the 18 A-scope items from docs/whisper-ecosystem/brief.md
into four phases (pre-emptive patches, engine abstraction, streaming
correctness, LLM guard) with stop-for-review boundaries between each.

Lists the new command + event contracts (#1 activeComputeDevice,
#14 list_gpus/set_preferred_gpu, #24 tentative segment flag, #28
parallel-mode toggle) that Workstream B will consume.

Co-authored-by: jars <jakejars@users.noreply.github.com>
2026-04-21 15:54:15 +01:00
23 changed files with 1171 additions and 118 deletions

View File

@@ -64,6 +64,8 @@ jobs:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
# System packages — same as check.yml but locked to release-build needs. # System packages — same as check.yml but locked to release-build needs.
# See check.yml for the per-package rationale (bindgen → libclang,
# llama-cpp-sys-2 vulkan feature → libvulkan / VULKAN_SDK).
- name: Install Linux deps - name: Install Linux deps
if: matrix.os == 'ubuntu-22.04' if: matrix.os == 'ubuntu-22.04'
run: | run: |
@@ -76,12 +78,28 @@ jobs:
libudev-dev \ libudev-dev \
patchelf \ patchelf \
cmake \ cmake \
build-essential build-essential \
libclang-dev \
clang \
libvulkan-dev \
glslang-tools \
spirv-tools
LIBCLANG_CANDIDATE=$(ls -d /usr/lib/llvm-*/lib 2>/dev/null | sort -V | tail -n1)
if [ -z "$LIBCLANG_CANDIDATE" ]; then
LIBCLANG_CANDIDATE=/usr/lib/x86_64-linux-gnu
fi
echo "LIBCLANG_PATH=$LIBCLANG_CANDIDATE" >> "$GITHUB_ENV"
- name: Install macOS deps - name: Install macOS deps
if: matrix.os == 'macos-latest' if: matrix.os == 'macos-latest'
run: | run: |
brew list cmake >/dev/null 2>&1 || brew install cmake brew list cmake >/dev/null 2>&1 || brew install cmake
brew list llvm >/dev/null 2>&1 || brew install llvm
brew install vulkan-headers vulkan-loader molten-vk shaderc
echo "LIBCLANG_PATH=$(brew --prefix llvm)/lib" >> "$GITHUB_ENV"
BREW_PREFIX=$(brew --prefix)
echo "VULKAN_SDK=$BREW_PREFIX" >> "$GITHUB_ENV"
echo "CMAKE_PREFIX_PATH=$BREW_PREFIX" >> "$GITHUB_ENV"
- name: Install Windows deps - name: Install Windows deps
if: matrix.os == 'windows-latest' if: matrix.os == 'windows-latest'
@@ -89,6 +107,14 @@ jobs:
run: | run: |
cmake --version cmake --version
choco install -y llvm --no-progress choco install -y llvm --no-progress
choco install -y vulkan-sdk --no-progress
$sdkRoot = Get-ChildItem -Directory "C:\VulkanSDK" | Sort-Object Name -Descending | Select-Object -First 1
if (-not $sdkRoot) {
Write-Error "VulkanSDK directory not found under C:\VulkanSDK after choco install"
exit 1
}
echo "VULKAN_SDK=$($sdkRoot.FullName)" >> $env:GITHUB_ENV
echo "LIBCLANG_PATH=C:\Program Files\LLVM\bin" >> $env:GITHUB_ENV
- name: Install Node - name: Install Node
uses: actions/setup-node@v4 uses: actions/setup-node@v4

View File

@@ -32,9 +32,15 @@ jobs:
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
# System packages whisper-rs-sys + Tauri need on each OS. # System packages whisper-rs-sys + llama-cpp-sys-2 + Tauri need on each OS.
# Linux is the heaviest because we depend on GTK/WebKit, audio, # - libclang-dev: bindgen (pulled by whisper-rs-sys + llama-cpp-sys-2)
# evdev, plus cmake for whisper.cpp. # needs a libclang shared library at build time.
# - Vulkan: llama-cpp-sys-2's `vulkan` feature wires `GGML_VULKAN=ON`
# for its embedded llama.cpp build, which runs `find_package(Vulkan)`
# and needs headers + loader + glslc at configure time (and
# libvulkan.so at link time). On Linux apt-get covers all four.
# - LIBCLANG_PATH: set explicitly because bindgen-0.72.1's default
# search path does not include /usr/lib/llvm-*/lib on 22.04.
- name: Install Linux deps - name: Install Linux deps
if: matrix.os == 'ubuntu-22.04' if: matrix.os == 'ubuntu-22.04'
run: | run: |
@@ -47,25 +53,61 @@ jobs:
libudev-dev \ libudev-dev \
patchelf \ patchelf \
cmake \ cmake \
build-essential build-essential \
libclang-dev \
clang \
libvulkan-dev \
glslang-tools \
spirv-tools
LIBCLANG_CANDIDATE=$(ls -d /usr/lib/llvm-*/lib 2>/dev/null | sort -V | tail -n1)
if [ -z "$LIBCLANG_CANDIDATE" ]; then
LIBCLANG_CANDIDATE=/usr/lib/x86_64-linux-gnu
fi
echo "LIBCLANG_PATH=$LIBCLANG_CANDIDATE" >> "$GITHUB_ENV"
# macOS: cmake is preinstalled in macos-latest but pin via brew to # macOS: cmake is preinstalled in macos-latest but pin via brew to
# be explicit and future-proof. # be explicit. Xcode CLT provides libclang but the runner's default
# clang install does not ship libclang.dylib in a discoverable
# location — use Homebrew's LLVM and point LIBCLANG_PATH at it.
#
# Vulkan on macOS is provided by MoltenVK (Vulkan → Metal shim).
# We install the Homebrew formulae individually rather than the
# LunarG macOS SDK, which ships as an interactive .dmg/.app and
# doesn't scriptify cleanly. shaderc gives us glslc, which
# find_package(Vulkan) requires at cmake configure time.
- name: Install macOS deps - name: Install macOS deps
if: matrix.os == 'macos-latest' if: matrix.os == 'macos-latest'
run: | run: |
brew list cmake >/dev/null 2>&1 || brew install cmake brew list cmake >/dev/null 2>&1 || brew install cmake
brew list llvm >/dev/null 2>&1 || brew install llvm
brew install vulkan-headers vulkan-loader molten-vk shaderc
echo "LIBCLANG_PATH=$(brew --prefix llvm)/lib" >> "$GITHUB_ENV"
BREW_PREFIX=$(brew --prefix)
echo "VULKAN_SDK=$BREW_PREFIX" >> "$GITHUB_ENV"
echo "CMAKE_PREFIX_PATH=$BREW_PREFIX" >> "$GITHUB_ENV"
# Windows: cmake + clang are needed by whisper-rs-sys. cmake is on # Windows: cmake + clang (for whisper-rs-sys/llama-cpp-sys bindgen)
# windows-latest by default; ensure it. # + Vulkan SDK (required by llama-cpp-sys-2 when the `vulkan`
# feature is on — it hard-panics on a missing VULKAN_SDK env var).
#
# choco's `vulkan-sdk` package installs into
# C:\VulkanSDK\<version>\; the canonical VULKAN_SDK path is that
# directory. We resolve it dynamically so a minor-version bump in
# the SDK doesn't hardcode-break this step.
- name: Install Windows deps - name: Install Windows deps
if: matrix.os == 'windows-latest' if: matrix.os == 'windows-latest'
shell: pwsh shell: pwsh
run: | run: |
# cmake is on the runner image; verify it.
cmake --version cmake --version
# LLVM/clang for whisper.cpp's sources.
choco install -y llvm --no-progress choco install -y llvm --no-progress
choco install -y vulkan-sdk --no-progress
$sdkRoot = Get-ChildItem -Directory "C:\VulkanSDK" | Sort-Object Name -Descending | Select-Object -First 1
if (-not $sdkRoot) {
Write-Error "VulkanSDK directory not found under C:\VulkanSDK after choco install"
exit 1
}
echo "VULKAN_SDK=$($sdkRoot.FullName)" >> $env:GITHUB_ENV
echo "LIBCLANG_PATH=C:\Program Files\LLVM\bin" >> $env:GITHUB_ENV
- name: Install Rust toolchain - name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable uses: dtolnay/rust-toolchain@stable

1
.gitignore vendored
View File

@@ -6,3 +6,4 @@ dist/
Cargo.lock Cargo.lock
.firecrawl/ .firecrawl/
.worktrees/ .worktrees/
.cargo/

View File

@@ -15,6 +15,70 @@ pub struct SystemProfile {
pub struct CpuInfo { pub struct CpuInfo {
pub logical_processors: usize, pub logical_processors: usize,
pub brand: String, pub brand: String,
pub features: CpuFeatures,
}
/// Runtime-detected CPU feature flags relevant to the speech-to-text
/// and LLM backends Kon ships. All whisper.cpp / llama.cpp / ggml
/// kernels degrade roughly two tiers without AVX2, which is why we
/// surface it separately: when AVX2 is absent, the UI should warn the
/// user that performance will be a fraction of what they would see
/// on a contemporary CPU. References:
/// - whisper-rs #8, #117 (illegal instruction on pre-AVX2 CPUs)
/// - Buzz FAQ (non-AVX2 fallback builds)
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub struct CpuFeatures {
pub avx2: bool,
pub avx512f: bool,
pub fma: bool,
pub sse4_2: bool,
pub neon: bool,
}
impl CpuFeatures {
/// Whether this CPU has the baseline ggml expects (AVX2 + FMA on
/// x86_64, NEON on aarch64). If false, the runtime banner fires.
pub fn has_ggml_baseline(&self) -> bool {
#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
{
return self.avx2 && self.fma;
}
#[cfg(target_arch = "aarch64")]
{
return self.neon;
}
#[allow(unreachable_code)]
false
}
}
/// Probes CPU feature flags via compile-time/runtime CPUID. On x86_64
/// we rely on `std::is_x86_feature_detected!`, which lowers to CPUID
/// at runtime. On aarch64 we assume NEON (architectural baseline);
/// on other targets all flags are false.
pub fn probe_cpu_features() -> CpuFeatures {
#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
{
return CpuFeatures {
avx2: std::is_x86_feature_detected!("avx2"),
avx512f: std::is_x86_feature_detected!("avx512f"),
fma: std::is_x86_feature_detected!("fma"),
sse4_2: std::is_x86_feature_detected!("sse4.2"),
neon: false,
};
}
#[cfg(target_arch = "aarch64")]
{
return CpuFeatures {
avx2: false,
avx512f: false,
fma: false,
sse4_2: false,
neon: true,
};
}
#[allow(unreachable_code)]
CpuFeatures::default()
} }
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
@@ -64,6 +128,7 @@ fn probe_cpu_from(sys: &System) -> CpuInfo {
.first() .first()
.map(|c| c.brand().to_string()) .map(|c| c.brand().to_string())
.unwrap_or_default(), .unwrap_or_default(),
features: probe_cpu_features(),
} }
} }
@@ -103,3 +168,53 @@ pub fn probe_system() -> SystemProfile {
os: probe_os(), os: probe_os(),
} }
} }
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn probe_cpu_features_runs_without_panicking() {
let _ = probe_cpu_features();
}
#[test]
fn probe_system_populates_cpu_features() {
let profile = probe_system();
// The check doesn't assume the runner has AVX2; it just asserts
// that the feature probe was actually called and is wired in.
let f = profile.cpu.features;
assert!(
f == f,
"CpuFeatures must be PartialEq so the runtime banner can debounce"
);
}
#[test]
fn ggml_baseline_matches_x86_64_rule() {
let features = CpuFeatures {
avx2: true,
fma: true,
..CpuFeatures::default()
};
// Only actually true on x86_64 — on other arches the helper
// returns false, which is equally fine for this test.
#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
assert!(features.has_ggml_baseline());
#[cfg(not(any(target_arch = "x86", target_arch = "x86_64")))]
assert!(!features.has_ggml_baseline());
}
#[test]
fn ggml_baseline_requires_both_avx2_and_fma() {
let features = CpuFeatures {
avx2: true,
fma: false,
..CpuFeatures::default()
};
#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
assert!(!features.has_ggml_baseline());
#[cfg(not(any(target_arch = "x86", target_arch = "x86_64")))]
assert!(!features.has_ggml_baseline());
}
}

View File

@@ -85,7 +85,7 @@ pub fn rank_recommendations(profile: &SystemProfile) -> Vec<ScoredModel> {
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
use crate::hardware::{CpuInfo, GpuAcceleration, GpuInfo, GpuVendor, Os}; use crate::hardware::{CpuFeatures, CpuInfo, GpuAcceleration, GpuInfo, GpuVendor, Os};
fn profile_with_ram(ram: Megabytes) -> SystemProfile { fn profile_with_ram(ram: Megabytes) -> SystemProfile {
SystemProfile { SystemProfile {
@@ -93,6 +93,7 @@ mod tests {
cpu: CpuInfo { cpu: CpuInfo {
logical_processors: 8, logical_processors: 8,
brand: "Test CPU".into(), brand: "Test CPU".into(),
features: CpuFeatures::default(),
}, },
gpu: None, gpu: None,
os: Os::Windows, os: Os::Windows,
@@ -105,6 +106,7 @@ mod tests {
cpu: CpuInfo { cpu: CpuInfo {
logical_processors: 8, logical_processors: 8,
brand: "Test CPU".into(), brand: "Test CPU".into(),
features: CpuFeatures::default(),
}, },
gpu: Some(GpuInfo { gpu: Some(GpuInfo {
vendor: GpuVendor::Nvidia, vendor: GpuVendor::Nvidia,

View File

@@ -3,6 +3,7 @@ name = "kon-transcription"
version = "0.1.0" version = "0.1.0"
edition = "2021" edition = "2021"
description = "Speech-to-text engine wrappers, model management, and inference concurrency for Kon" description = "Speech-to-text engine wrappers, model management, and inference concurrency for Kon"
build = "build.rs"
[dependencies] [dependencies]
kon-core = { path = "../core" } kon-core = { path = "../core" }
@@ -28,3 +29,8 @@ thiserror = "2"
# Structured logging at backend boundaries (observability for initial_prompt flow). # Structured logging at backend boundaries (observability for initial_prompt flow).
tracing = "0.1" tracing = "0.1"
[dev-dependencies]
# TcpListener fixture for the download resume tests (mirrors kon-llm).
tokio = { version = "1", features = ["rt", "sync", "net", "io-util", "macros"] }
tempfile = "3"

View File

@@ -0,0 +1,73 @@
//! Build-time guard for item #6 of the Whisper ecosystem pass.
//!
//! On Windows, linking `whisper-rs-sys` (MSVC C++ runtime) and the
//! `tokenizers` crate (which pulls a different MSVC CRT via its
//! onnxruntime + Rust-side dependencies) in the same binary has been a
//! repeated failure mode — most recently Whispering v7.11.0 shipped a
//! broken Windows build over exactly this conflict. Reference:
//! https://github.com/EpicenterHQ/epicenter/releases/tag/v7.11.0
//!
//! The easiest defence is to refuse to compile at all if any part of the
//! workspace ever pulls `tokenizers` into the dependency graph on a
//! Windows target. If we ever legitimately need it we can reintroduce
//! it via a sidecar (isolated process, separate CRT) rather than
//! linking it into `kon_lib`.
//!
//! The check is advisory on non-Windows targets — it still prints a
//! cargo:warning if `tokenizers` appears, so the Windows failure isn't
//! a surprise at CI time when we build cross-platform from Linux.
use std::env;
use std::fs;
use std::path::PathBuf;
fn main() {
println!("cargo:rerun-if-changed=build.rs");
let target_os = env::var("CARGO_CFG_TARGET_OS").unwrap_or_default();
let manifest_dir = PathBuf::from(env::var("CARGO_MANIFEST_DIR").unwrap_or_else(|_| ".".into()));
// Walk up to workspace root: crates/transcription/ -> crates/ -> root
let workspace_root = manifest_dir
.ancestors()
.find(|p| p.join("Cargo.lock").exists())
.map(PathBuf::from);
let Some(root) = workspace_root else {
// No lockfile yet (e.g. first-ever cargo run). Nothing to check.
return;
};
let lock_path = root.join("Cargo.lock");
println!("cargo:rerun-if-changed={}", lock_path.display());
let lock = match fs::read_to_string(&lock_path) {
Ok(s) => s,
Err(_) => return,
};
let has_tokenizers = lock
.lines()
.any(|line| matches!(line.trim(), "name = \"tokenizers\""));
if !has_tokenizers {
return;
}
if target_os == "windows" {
panic!(
"kon-transcription: the `tokenizers` crate appears in Cargo.lock and this is a \
Windows build. Linking `whisper-rs-sys` + `tokenizers` in the same binary has \
been a persistent MSVC C-runtime conflict (see Whispering v7.11.0). Route any \
tokenizer usage through an out-of-process sidecar instead, or gate it off for \
Windows. Brief item #6."
);
}
println!(
"cargo:warning=kon-transcription: `tokenizers` crate is in the dependency graph. \
This build is non-Windows so the link will succeed, but Windows builds will panic \
at build time per docs/whisper-ecosystem/brief.md item #6. Isolate tokenizer usage \
in a sidecar before a Windows ship."
);
}

View File

@@ -52,6 +52,11 @@ pub fn list_downloaded() -> Vec<ModelId> {
/// Download all files for a model, calling the progress callback per chunk. /// Download all files for a model, calling the progress callback per chunk.
/// Files are downloaded to a .part suffix and atomically renamed on completion. /// Files are downloaded to a .part suffix and atomically renamed on completion.
///
/// For files that declare a `sha256` checksum we validate an existing
/// complete file before skipping the download — a truncated or
/// tampered file gets redownloaded automatically (pattern ported from
/// `kon-llm`'s model_manager, item #8 in the Whisper ecosystem brief).
pub async fn download( pub async fn download(
id: &ModelId, id: &ModelId,
progress: impl Fn(DownloadProgress) + Send + 'static, progress: impl Fn(DownloadProgress) + Send + 'static,
@@ -64,7 +69,29 @@ pub async fn download(
for file in &entry.files { for file in &entry.files {
let dest = dir.join(file.filename); let dest = dir.join(file.filename);
if dest.exists() { if dest.exists() {
continue; if let Some(expected_sha) = file.sha256 {
// Validate the existing file. If the hash doesn't match,
// the file is corrupt (partial download, tampering, bit
// rot) and we must re-fetch it to avoid crashing on
// model load later.
match sha256_of_file(&dest) {
Ok(actual) if actual.eq_ignore_ascii_case(expected_sha) => continue,
Ok(_actual) => {
// Corrupt — remove + fall through to re-download.
let _ = std::fs::remove_file(&dest);
}
Err(e) => {
return Err(KonError::DownloadFailed(format!(
"failed to verify existing {}: {e}",
file.filename
)));
}
}
} else {
// No checksum — honour the existing file as-is; the
// engine will barf on load if it's broken.
continue;
}
} }
download_file(file, &dest, id, &progress).await?; download_file(file, &dest, id, &progress).await?;
} }
@@ -72,6 +99,24 @@ pub async fn download(
Ok(()) Ok(())
} }
/// Non-streaming SHA256 of a file on disk. Used by `download()` to
/// validate an existing complete file before trusting it.
fn sha256_of_file(path: &Path) -> std::io::Result<String> {
use sha2::{Digest, Sha256};
let mut hasher = Sha256::new();
let mut file = std::fs::File::open(path)?;
let mut buffer = [0u8; 8192];
loop {
let n = std::io::Read::read(&mut file, &mut buffer)?;
if n == 0 {
break;
}
hasher.update(&buffer[..n]);
}
Ok(format!("{:x}", hasher.finalize()))
}
/// Download a single file with HTTP Range resume and optional SHA256 verification. /// Download a single file with HTTP Range resume and optional SHA256 verification.
/// ///
/// Resume pattern from Buzz (chidiwilliams/buzz): if a .part file exists, /// Resume pattern from Buzz (chidiwilliams/buzz): if a .part file exists,
@@ -118,8 +163,28 @@ async fn download_file(
.await .await
.map_err(|e| KonError::DownloadFailed(e.to_string()))?; .map_err(|e| KonError::DownloadFailed(e.to_string()))?;
// Check if server supports range (206 Partial Content) or gave full file (200) // If we requested Range but the server returned 200 (full file), the
let actually_resuming = resuming && response.status().as_u16() == 206; // server does not support resume. Rather than blindly appending a
// full file on top of our partial bytes (which would produce a
// corrupt result), restart cleanly. This mirrors the kon-llm
// ResumeUnsupported branch — item #8 of the brief.
let actually_resuming = if resuming {
match response.status().as_u16() {
206 => true,
200 => {
// Server ignored our Range header — treat as fresh start.
// The old .part bytes are discarded below.
false
}
other => {
return Err(KonError::DownloadFailed(format!(
"resume request returned unexpected status {other}"
)));
}
}
} else {
false
};
let total_bytes = if actually_resuming { let total_bytes = if actually_resuming {
// Content-Range: bytes START-END/TOTAL — extract TOTAL // Content-Range: bytes START-END/TOTAL — extract TOTAL
@@ -202,6 +267,10 @@ async fn download_file(
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
use sha2::Digest;
use tempfile::tempdir;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpListener;
#[test] #[test]
fn model_dir_returns_correct_path() { fn model_dir_returns_correct_path() {
@@ -223,4 +292,124 @@ mod tests {
// This just verifies the function doesn't panic // This just verifies the function doesn't panic
assert!(list.len() <= kon_core::model_registry::all_models().len()); assert!(list.len() <= kon_core::model_registry::all_models().len());
} }
#[test]
fn sha256_of_file_matches_sha2() {
let dir = tempdir().unwrap();
let path = dir.path().join("f.bin");
std::fs::write(&path, b"hello world").unwrap();
let expected = format!("{:x}", sha2::Sha256::digest(b"hello world"));
assert_eq!(sha256_of_file(&path).unwrap(), expected);
}
/// A minimal HTTP server that sends a Range response when a Range
/// header is present and otherwise sends the full body. Ported from
/// crates/llm/src/model_manager.rs to give the transcription
/// download stack the same fixture-backed coverage.
async fn spawn_range_server(content: Vec<u8>) -> std::net::SocketAddr {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
let (mut socket, _) = listener.accept().await.unwrap();
let mut buf = vec![0u8; 2048];
let size = socket.read(&mut buf).await.unwrap();
let request = String::from_utf8_lossy(&buf[..size]).to_lowercase();
let range_start = request
.lines()
.find_map(|line| line.strip_prefix("range: bytes="))
.and_then(|line| line.strip_suffix('-'))
.and_then(|line| line.trim().parse::<usize>().ok());
if let Some(start) = range_start {
let body = &content[start..];
let response = format!(
"HTTP/1.1 206 Partial Content\r\n\
Content-Length: {}\r\n\
Content-Range: bytes {}-{}/{}\r\n\
Accept-Ranges: bytes\r\n\r\n",
body.len(),
start,
content.len() - 1,
content.len(),
);
socket.write_all(response.as_bytes()).await.unwrap();
socket.write_all(body).await.unwrap();
} else {
let response = format!(
"HTTP/1.1 200 OK\r\n\
Content-Length: {}\r\n\
Accept-Ranges: bytes\r\n\r\n",
content.len(),
);
socket.write_all(response.as_bytes()).await.unwrap();
socket.write_all(&content).await.unwrap();
}
});
addr
}
/// ModelFile stores `&'static str` fields, so we leak the strings
/// once per test — tests are one-shot, so the cost is noise.
fn leak(s: String) -> &'static str {
Box::leak(s.into_boxed_str())
}
#[tokio::test]
async fn download_file_resumes_from_partial_and_verifies_sha() {
let body = b"resumable transcription payload".to_vec();
let expected_sha = format!("{:x}", sha2::Sha256::digest(&body));
let addr = spawn_range_server(body.clone()).await;
let dir = tempdir().unwrap();
let dest = dir.path().join("fixture.bin");
let part = dest.with_extension("bin.part");
// Pretend we already downloaded the first 7 bytes.
std::fs::write(&part, &body[..7]).unwrap();
let file = ModelFile {
filename: leak(dest.file_name().unwrap().to_string_lossy().into_owned()),
url: leak(format!("http://{addr}/fixture.bin")),
size: kon_core::types::Megabytes(0),
sha256: None, // resume path only kicks in when sha256 is absent
};
let id = ModelId::new("test-fixture");
download_file(&file, &dest, &id, &|_| ()).await.unwrap();
let bytes = std::fs::read(&dest).unwrap();
assert_eq!(bytes, body);
assert!(!part.exists());
// Confirm the full file hash matches what we would have got via
// a clean download — gives the resume path indirect integrity
// coverage even when the ModelFile has no sha256 set.
assert_eq!(sha256_of_file(&dest).unwrap(), expected_sha);
}
#[tokio::test]
async fn download_file_fails_on_sha_mismatch_and_cleans_part_file() {
let body = b"speech-to-text fixture body".to_vec();
let addr = spawn_range_server(body.clone()).await;
let dir = tempdir().unwrap();
let dest = dir.path().join("fixture.bin");
let file = ModelFile {
filename: leak(dest.file_name().unwrap().to_string_lossy().into_owned()),
url: leak(format!("http://{addr}/fixture.bin")),
size: kon_core::types::Megabytes(0),
sha256: Some(leak("deadbeef".repeat(8))),
};
let id = ModelId::new("test-fixture");
let err = download_file(&file, &dest, &id, &|_| ())
.await
.expect_err("mismatched sha must fail");
let msg = err.to_string();
assert!(msg.contains("SHA256 mismatch"), "unexpected error: {msg}");
assert!(!dest.exists(), ".part → dest rename must not run on mismatch");
let part = dest.with_extension("bin.part");
assert!(!part.exists(), "failed hash must clean up the .part file");
}
} }

View File

@@ -0,0 +1,270 @@
# Workstream A — Systems hardening + streaming correctness
*Execution plan for the Codex half of the Whisper-ecosystem pass.*
*Branch: `phase4-systems-f7d0` (base: `main`). Companion: `phase4-ux-f7d0` (Workstream B).*
This is the backend / systems half. Every task below maps to an atomic
item in `docs/whisper-ecosystem/brief.md`. Items #3, #4 (UX side only),
#5, #10, #11, #14 (UI), #15, #16, #17, #20, #27, #30, #31 are owned by
Workstream B and are listed here only where the contract is shared.
---
## Scope recap
**In scope for A:** items #1, #2, #6, #7, #8, #9, #12, #13, #18, #19,
#21, #22, #23, #24, #25, #26, #28, #29.
**Explicit skip:** #18 (initial-prompt priming) is already shipped —
`src-tauri/src/commands/mod.rs::build_initial_prompt` plus
`crates/transcription/src/whisper_rs_backend.rs` already wire the
request prompt + profile prompt + profile vocabulary into
`FullParams::set_initial_prompt`. Workstream A verifies via the existing
lib tests and moves on; no new code.
**Untouchable (owned by B):** `src/lib/pages/*.svelte`, `src/routes/**`,
`crates/ai-formatting/src/llm_client.rs` (B may only edit
`CLEANUP_PROMPT` on that file).
**Untouchable (global):** the `ggml` dedup workaround in
`src-tauri/build.rs` is pinned as a known interim — do not touch.
---
## Execution order
The backlog groups itself into four natural phases. Each phase lands as
a sequence of small, one-concern commits, with `cargo test --workspace
--lib && cargo build -p kon && npm run check` green at every commit
boundary. At the end of each phase we pause for review before the next.
### Phase A.1 — Pre-emptive patches (no new UX surface)
| Seq | Item | Effect | Depends on |
|---|---|---|---|
| 1 | **#2**: pre-approve `http://127.0.0.1:*` in Tauri capabilities for LLM | Adds `http:default-scope-127.0.0.1` style capability + widens `connect-src` CSP to localhost | — |
| 2 | **#6**: guard `whisper-rs-sys` + `tokenizers` on Windows | Audit: Kon doesn't link `tokenizers` directly; we add a `#[cfg(all(target_os = "windows"))]` compile-time assert in `kon-transcription/build.rs` that fails the build if anything in the tree pulls in `tokenizers` on Windows | — |
| 3 | **#7**: CPU feature detection + non-AVX2 fallback surface | Extend `crates/core/src/hardware.rs` with `CpuFeatures` (avx2, avx512, fma, sse4_2) via `std::is_x86_feature_detected!` at runtime; surface as part of `RuntimeCapabilities`; when AVX2 is absent, emit a `warning` log + `runtime-warning` event so the frontend can show a banner | #1 |
| 4 | **#1**: detect and surface active compute device | Extend `RuntimeCapabilities` with an `ActiveComputeDevice` struct (`kind: "cuda" \| "vulkan" \| "metal" \| "cpu"`, `label: String`, `reason: Option<String>`). For Whisper, read the first `whisper_print_system_info` line via a shim in `whisper_rs_backend`; reason is filled on any CPU fallback (e.g., "Vulkan loader not found") | #7 |
| 5 | **#8**: checksum + resumable model downloads; retain audio on failure | Port the `crates/llm/src/model_manager.rs` pattern (SHA256 verify, `.part` file resume, `ResumeUnsupported` error) into `crates/transcription/src/model_manager.rs::download_file`. The existing code already does incremental SHA256 and range resume but lacks (a) validation of an **existing full file** before re-downloading, (b) ResumeUnsupported signalling, (c) test coverage parity. Retaining audio on failure is already covered in `commands/live.rs` via `save_audio` — add an explicit retry-friendly error classification so the frontend can render "Retry transcription" (a B surface, but the error enum ships here) | — |
| 6 | **#9**: disable macOS App Nap during capture + LLM | Add `commands/power.rs` with `cfg(target_os = "macos")` using `NSProcessInfo beginActivityWithOptions:reason:` to pin a power-assertion handle during live sessions and LLM generation | — |
| 7 | **#12**: bundle Vulkan loader + libssl on Windows, CPU fallback | Update `src-tauri/tauri.conf.json` bundle `resources` to list `vulkan-1.dll` and `libssl-3-x64.dll`/`libcrypto-3-x64.dll`; add a `cfg(target_os = "windows")` startup hook that probes `LoadLibraryW("vulkan-1.dll")` and downgrades `active_compute_device` to CPU with a clear reason if absent. Ship a pre-`cargo build` note in CI (README) — we cannot sign the installer from Linux CI | #1 |
**Commit boundary:** `cargo test --workspace --lib` (must include new
tests for checksum resume + CPU feature detect + active-device shim),
`cargo build -p kon`, `npm run check`. Then **stop for review.**
### Phase A.2 — Engine abstraction (#13, #19)
| Seq | Item | Effect | Depends on |
|---|---|---|---|
| 8 | **#13 (engine trait)**: `kon-transcription` already has `LocalEngine` and a `SpeechBackend` enum over `transcribe-rs` adapter + `WhisperRsBackend`. This is a 90% version of Handy's `transcribe-rs` pattern. We lift it one more inch: replace the enum with a public `Transcriber` trait (`load`, `transcribe_sync`, `capabilities`) + blanket impls for the existing two backends, and gate `WhisperRsBackend` behind the `whisper` feature so non-Whisper builds compile without pulling `whisper-rs-sys` | — |
| 9 | **#19**: progressive WAV write during live capture | Extend `commands/live.rs` to stream captured mono-16kHz samples directly into a `.wav` in the session folder via `kon_audio::WavWriter::append`. Replace the in-memory `kept_audio: Vec<f32>` when `save_audio` is on with a disk-backed writer; on crash, the partial file is a playable WAV. Commits the `commands/live.rs` frame-size bookkeeping untouched | #13 (uses `Transcriber::capabilities().sample_rate`) |
**Commit boundary:** tests (`transcriber_trait_is_object_safe`,
`wav_writer_survives_crash`), build, check. **Stop for review.**
### Phase A.3 — Streaming correctness (#21#26)
This is the biggest change surface. We replace the custom RMS speech
gate in `commands/live.rs` with a Silero-VAD-gated chunker and layer
hallucination defences on top. Everything goes into a new
`crates/transcription/src/streaming/` module; `commands/live.rs`
becomes a thin driver.
| Seq | Item | Effect | Depends on |
|---|---|---|---|
| 10 | **#23**: warm-up silent WAV at app launch and after 60s idle | Add a tiny 1s silent-WAV loader (shipped in `src-tauri/assets/warmup.wav`) that runs through `LocalEngine::transcribe_sync` once after load_model; reschedules itself after any 60s idle gap in live sessions. Fixes the 45s cold-start latency cliff | #13 |
| 11 | **#21**: Silero-VAD-gated chunker | Add `silero-vad` (ort-backed, small ONNX model) as a dependency. New `streaming::VadChunker` takes a 16 kHz mono stream, emits `(chunk_start_sample, samples)` pairs when the VAD score crosses a threshold with hysteresis. Configurable threshold (default 0.5, hysteresis 0.35). Replaces the RMS-based `evaluate_speech_gate` for the chunk-boundary decision. Current `evaluate_speech_gate` stays as a _second-line_ silence skip | #19 (buffer ownership moved) |
| 12 | **#22**: hallucination blocklist + `avg_logprob`/`no_speech_prob` gate | Extend `WhisperRsBackend` to surface `no_speech_prob` + `avg_logprob` per segment (currently discarded). Add `streaming::HallucinationFilter` with a blocklist (static + user-editable via profile terms — blocked list, not vocab) and a confidence gate. Drop whole chunks that score below gate; drop segments whose text matches blocklist phrases | #21 |
| 13 | **#24**: LocalAgreement-n commit policy | Add `streaming::CommitPolicy::LocalAgreement { n: 2 }`. Two consecutive passes must produce matching prefix tokens before emission; tentative tail is sent with a `tentative: true` flag on `LiveResultMessage.segments` for the UI (B renders the dashed underline — contract below) | #22 |
| 14 | **#25**: aggressive buffer trim tied to commit points | Replace the current "drain OVERLAP_SAMPLES" with "drain everything up to the last commit point emitted by `CommitPolicy`". Guarantees the working buffer stays bounded regardless of wall-clock session length | #24 |
| 15 | **#26**: repetition detector + context window reset | Add `streaming::RepetitionDetector`: three consecutive identical token runs (per whisper-rs token, or per-word if running Parakeet) triggers `WhisperContext::create_state` + drop the offending chunk, logs a `runtime-warning` event | #22 |
**Commit boundary:** tests for the VAD chunker (with a fixture WAV that
contains silence → speech → silence), the LocalAgreement commit policy
(feed two fake passes, assert only the agreed prefix emits), and the
repetition detector. **Stop for review.**
### Phase A.4 — LLM guard (#28, #29)
| Seq | Item | Effect | Depends on |
|---|---|---|---|
| 16 | **#28**: sequential Whisper→LLM execution; single-GPU guard | Add `crates/transcription/src/concurrency.rs::GpuGuard` (already exists as file; extend it) — an async `Semaphore::new(1)` gate that wraps every `transcribe_sync` and every `LlmEngine::generate` on single-GPU systems. A `parallel_mode: bool` setting (default false; exposed via `get_runtime_capabilities` for B to wire into Settings) opens the gate to `n = 2` for users with ≥16 GB VRAM detected by `probe_gpu` | #1 |
| 17 | **#29**: plain-text pre-formatter before LLM prompt | `crates/ai-formatting/src/pipeline.rs` already joins segments; extend it to strip timestamps/IDs and normalise whitespace for the LLM call. New `crates/ai-formatting/src/to_plain_text.rs` with a pure function + tests | — |
**Commit boundary:** tests, build, check. **Stop for review.**
---
## Functions extended vs replaced
| Existing function / module | Extend or replace | Rationale |
|---|---|---|
| `commands::mod::build_initial_prompt` | **Keep as-is** | Already covers #18 |
| `commands::models::get_runtime_capabilities` | **Extend** | Add `active_compute_device`, `cpu_features`, `parallel_mode_available` fields |
| `commands::live::evaluate_speech_gate` | **Replace** with VAD-gated chunker, keep RMS version as a cheap fallback when VAD ONNX model missing |
| `commands::live::run_live_session` | **Extend** | Switch buffer ownership to `streaming::StreamingSession`, keep the `cpal` plumbing |
| `crates/transcription/src/model_manager::download_file` | **Extend** | Harden to match `kon-llm`'s resume + sha behaviour, add `ResumeUnsupported` arm |
| `crates/transcription/src/whisper_rs_backend::transcribe_sync` | **Extend** | Surface `no_speech_prob` + `avg_logprob` per segment |
| `crates/transcription/src/local_engine::SpeechBackend` | **Replace** with public `Transcriber` trait |
| `crates/transcription/src/concurrency` | **Extend** | Already a stub; add GPU guard wrapper |
| `crates/ai-formatting/src/pipeline` | **Extend** only the plain-text join path |
| `crates/core/src/hardware::probe_system` | **Extend** | Add `cpu_features` + flesh out `probe_gpu` (which currently returns `None`) with a best-effort Vulkan loader probe |
---
## New commands and event contracts (for Workstream B)
Workstream B needs stable surfaces to wire into. These are the only
Rust-side contracts A commits to in Phase A.1 and A.2 (Phase A.3/A.4
contracts are listed inline above):
### Item #1: active compute device
Extend `RuntimeCapabilities` (no new command):
```jsonc
{
"accelerators": ["cpu", "vulkan"],
"activeComputeDevice": {
"kind": "vulkan", // "cuda" | "vulkan" | "metal" | "cpu"
"label": "NVIDIA GeForce RTX 3060 (Vulkan)",
"reason": null // or "Vulkan loader not found, running on CPU"
},
"cpuFeatures": {
"avx2": true,
"avx512": false,
"fma": true,
"sse4_2": true
},
"parallelModeAvailable": false, // true only when >= 16 GB VRAM
"engines": [ /* unchanged */ ]
}
```
Event:
```
event: runtime-warning
payload: { kind: "avx2-missing" | "vulkan-loader-missing" | "cuda-fallback", message: string }
```
Emitted once at startup; B shows a dismissible Settings banner.
### Item #14: GPU enumeration (for B's explicit selector)
New command, for B to wire the Settings dropdown:
```
command: list_gpus()
returns: [
{
"id": "cuda:0",
"label": "NVIDIA GeForce RTX 3060",
"vram_mb": 12288,
"backend": "cuda", // "cuda" | "vulkan" | "metal" | "cpu"
"active": true
},
{
"id": "cpu",
"label": "CPU (Ryzen 7 7840U, 16 threads)",
"vram_mb": null,
"backend": "cpu",
"active": false
}
]
command: set_preferred_gpu(id: string)
returns: Ok(()) — persists to settings table, takes effect on next model load
```
Note that **actually swapping** the device requires rebuilding the
whisper-cpp backend with a different feature flag; for now this command
stores the preference, surfaces a "Restart to apply" toast, and
`prewarm_default_model` re-checks it on next launch.
### Item #19 / #23: warm-up and progressive WAV
No new frontend-visible command. The `save_audio` flag on
`start_live_transcription_session` already covers this; B should
continue to use it.
### Item #24: tentative vs committed segments (streaming)
`LiveResultMessage.segments[i]` gets a new field:
```
{
"start": 1.2,
"end": 1.8,
"text": "hello world",
"tentative": false // true when emitted under LocalAgreement tail
}
```
B is expected to render `tentative: true` with a dashed underline in
the preview overlay. Existing consumers that ignore unknown fields keep
working; this field is additive.
### Item #28: parallel-mode setting
Extend `SettingsState` on B's side to include
`parallelWhisperLlm: boolean` (default `false`). B wires this to a new
Rust command:
```
command: set_parallel_gpu_mode(enabled: bool) -> Ok(())
```
which flips the `GpuGuard`'s semaphore permit count. Gated in Settings
behind `runtimeCapabilities.parallelModeAvailable`.
---
## Test strategy
- Every phase ends green on `cargo test --workspace --lib && cargo
build -p kon && npm run check`.
- `phase4-systems-f7d0` never regresses the 116 existing lib tests. If
a test starts failing and the cause is an A change, it's fixed
in-phase, not deferred.
- New unit tests land with their phase: the VAD chunker has a WAV
fixture test, the LocalAgreement policy has a synthetic two-pass
test, the download resume already has a `tokio::net::TcpListener`
fixture test — we port the pattern.
- Manual smoke test after Phase A.3 on Jake's dev box: 10-minute live
dictation session, assert (a) no latency growth past chunk 30, (b)
no "Thanks for watching" in output, (c) raw WAV playable after
force-kill.
---
## Known unknowns / risks
- **`silero-vad` crate choice.** There's no first-party Rust binding;
the pragmatic options are `voice_activity_detector` (ort-backed,
pulls `ort` which is large) or porting Handy's direct `ort` call.
Decision point in Phase A.3; if the dep cost is unacceptable we fall
back to the existing RMS gate with stricter thresholds and document
the gap. Either way the `VadChunker` trait is the same, so the
downstream `StreamingSession` code doesn't change.
- **macOS App Nap** (#9): we cannot CI this on Linux. Ships with the
code pattern from Whispering's source and a manual-test-only note.
- **Windows installer Vulkan bundling** (#12): same — ships with
`tauri.conf.json` changes + a README note, verified manually on next
Windows build.
- **`build.rs` compile-time assert** (#6): if `cargo metadata` reports
`tokenizers` in any tree position on Windows, we `println!("cargo:
warning=...")` + `panic!` out of the build. Easier to read than a
runtime crash on first model load.
---
## Commit conventions on this branch
- One concern per commit. Subject line `feat(A.2):`, `fix(A.1):`, etc.,
referencing the phase so the reviewer can group them.
- Commit message body states which brief-item the commit closes.
- `cargo test --workspace --lib && cargo build -p kon && npm run check`
green before every commit, per the rules of engagement.

85
package-lock.json generated
View File

@@ -9,6 +9,7 @@
"version": "0.1.0", "version": "0.1.0",
"license": "MIT", "license": "MIT",
"dependencies": { "dependencies": {
"@chenglou/pretext": "0.0.5",
"@tauri-apps/api": "^2", "@tauri-apps/api": "^2",
"@tauri-apps/plugin-dialog": "^2.6.0", "@tauri-apps/plugin-dialog": "^2.6.0",
"@tauri-apps/plugin-global-shortcut": "^2.3.1", "@tauri-apps/plugin-global-shortcut": "^2.3.1",
@@ -29,6 +30,12 @@
"vite": "^6.0.3" "vite": "^6.0.3"
} }
}, },
"node_modules/@chenglou/pretext": {
"version": "0.0.5",
"resolved": "https://registry.npmjs.org/@chenglou/pretext/-/pretext-0.0.5.tgz",
"integrity": "sha512-A8GZN10REdFGsyuiUgLV8jjPDDFMg5GmgxGWV0I3igxBOnzj+jgz2VMmVD7g+SFyoctfeqHFxbNatKSzVRWtRg==",
"license": "MIT"
},
"node_modules/@esbuild/aix-ppc64": { "node_modules/@esbuild/aix-ppc64": {
"version": "0.25.12", "version": "0.25.12",
"resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.25.12.tgz", "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.25.12.tgz",
@@ -666,9 +673,6 @@
"arm" "arm"
], ],
"dev": true, "dev": true,
"libc": [
"glibc"
],
"license": "MIT", "license": "MIT",
"optional": true, "optional": true,
"os": [ "os": [
@@ -683,9 +687,6 @@
"arm" "arm"
], ],
"dev": true, "dev": true,
"libc": [
"musl"
],
"license": "MIT", "license": "MIT",
"optional": true, "optional": true,
"os": [ "os": [
@@ -700,9 +701,6 @@
"arm64" "arm64"
], ],
"dev": true, "dev": true,
"libc": [
"glibc"
],
"license": "MIT", "license": "MIT",
"optional": true, "optional": true,
"os": [ "os": [
@@ -717,9 +715,6 @@
"arm64" "arm64"
], ],
"dev": true, "dev": true,
"libc": [
"musl"
],
"license": "MIT", "license": "MIT",
"optional": true, "optional": true,
"os": [ "os": [
@@ -734,9 +729,6 @@
"loong64" "loong64"
], ],
"dev": true, "dev": true,
"libc": [
"glibc"
],
"license": "MIT", "license": "MIT",
"optional": true, "optional": true,
"os": [ "os": [
@@ -751,9 +743,6 @@
"loong64" "loong64"
], ],
"dev": true, "dev": true,
"libc": [
"musl"
],
"license": "MIT", "license": "MIT",
"optional": true, "optional": true,
"os": [ "os": [
@@ -768,9 +757,6 @@
"ppc64" "ppc64"
], ],
"dev": true, "dev": true,
"libc": [
"glibc"
],
"license": "MIT", "license": "MIT",
"optional": true, "optional": true,
"os": [ "os": [
@@ -785,9 +771,6 @@
"ppc64" "ppc64"
], ],
"dev": true, "dev": true,
"libc": [
"musl"
],
"license": "MIT", "license": "MIT",
"optional": true, "optional": true,
"os": [ "os": [
@@ -802,9 +785,6 @@
"riscv64" "riscv64"
], ],
"dev": true, "dev": true,
"libc": [
"glibc"
],
"license": "MIT", "license": "MIT",
"optional": true, "optional": true,
"os": [ "os": [
@@ -819,9 +799,6 @@
"riscv64" "riscv64"
], ],
"dev": true, "dev": true,
"libc": [
"musl"
],
"license": "MIT", "license": "MIT",
"optional": true, "optional": true,
"os": [ "os": [
@@ -836,9 +813,6 @@
"s390x" "s390x"
], ],
"dev": true, "dev": true,
"libc": [
"glibc"
],
"license": "MIT", "license": "MIT",
"optional": true, "optional": true,
"os": [ "os": [
@@ -853,9 +827,6 @@
"x64" "x64"
], ],
"dev": true, "dev": true,
"libc": [
"glibc"
],
"license": "MIT", "license": "MIT",
"optional": true, "optional": true,
"os": [ "os": [
@@ -870,9 +841,6 @@
"x64" "x64"
], ],
"dev": true, "dev": true,
"libc": [
"musl"
],
"license": "MIT", "license": "MIT",
"optional": true, "optional": true,
"os": [ "os": [
@@ -1204,9 +1172,6 @@
"arm64" "arm64"
], ],
"dev": true, "dev": true,
"libc": [
"glibc"
],
"license": "MIT", "license": "MIT",
"optional": true, "optional": true,
"os": [ "os": [
@@ -1224,9 +1189,6 @@
"arm64" "arm64"
], ],
"dev": true, "dev": true,
"libc": [
"musl"
],
"license": "MIT", "license": "MIT",
"optional": true, "optional": true,
"os": [ "os": [
@@ -1244,9 +1206,6 @@
"x64" "x64"
], ],
"dev": true, "dev": true,
"libc": [
"glibc"
],
"license": "MIT", "license": "MIT",
"optional": true, "optional": true,
"os": [ "os": [
@@ -1264,9 +1223,6 @@
"x64" "x64"
], ],
"dev": true, "dev": true,
"libc": [
"musl"
],
"license": "MIT", "license": "MIT",
"optional": true, "optional": true,
"os": [ "os": [
@@ -1454,9 +1410,6 @@
"arm64" "arm64"
], ],
"dev": true, "dev": true,
"libc": [
"glibc"
],
"license": "Apache-2.0 OR MIT", "license": "Apache-2.0 OR MIT",
"optional": true, "optional": true,
"os": [ "os": [
@@ -1474,9 +1427,6 @@
"arm64" "arm64"
], ],
"dev": true, "dev": true,
"libc": [
"musl"
],
"license": "Apache-2.0 OR MIT", "license": "Apache-2.0 OR MIT",
"optional": true, "optional": true,
"os": [ "os": [
@@ -1494,9 +1444,6 @@
"riscv64" "riscv64"
], ],
"dev": true, "dev": true,
"libc": [
"glibc"
],
"license": "Apache-2.0 OR MIT", "license": "Apache-2.0 OR MIT",
"optional": true, "optional": true,
"os": [ "os": [
@@ -1514,9 +1461,6 @@
"x64" "x64"
], ],
"dev": true, "dev": true,
"libc": [
"glibc"
],
"license": "Apache-2.0 OR MIT", "license": "Apache-2.0 OR MIT",
"optional": true, "optional": true,
"os": [ "os": [
@@ -1534,9 +1478,6 @@
"x64" "x64"
], ],
"dev": true, "dev": true,
"libc": [
"musl"
],
"license": "Apache-2.0 OR MIT", "license": "Apache-2.0 OR MIT",
"optional": true, "optional": true,
"os": [ "os": [
@@ -2205,9 +2146,6 @@
"arm64" "arm64"
], ],
"dev": true, "dev": true,
"libc": [
"glibc"
],
"license": "MPL-2.0", "license": "MPL-2.0",
"optional": true, "optional": true,
"os": [ "os": [
@@ -2229,9 +2167,6 @@
"arm64" "arm64"
], ],
"dev": true, "dev": true,
"libc": [
"musl"
],
"license": "MPL-2.0", "license": "MPL-2.0",
"optional": true, "optional": true,
"os": [ "os": [
@@ -2253,9 +2188,6 @@
"x64" "x64"
], ],
"dev": true, "dev": true,
"libc": [
"glibc"
],
"license": "MPL-2.0", "license": "MPL-2.0",
"optional": true, "optional": true,
"os": [ "os": [
@@ -2277,9 +2209,6 @@
"x64" "x64"
], ],
"dev": true, "dev": true,
"libc": [
"musl"
],
"license": "MPL-2.0", "license": "MPL-2.0",
"optional": true, "optional": true,
"os": [ "os": [

View File

@@ -14,6 +14,7 @@
}, },
"license": "MIT", "license": "MIT",
"dependencies": { "dependencies": {
"@chenglou/pretext": "0.0.5",
"@tauri-apps/api": "^2", "@tauri-apps/api": "^2",
"@tauri-apps/plugin-dialog": "^2.6.0", "@tauri-apps/plugin-dialog": "^2.6.0",
"@tauri-apps/plugin-global-shortcut": "^2.3.1", "@tauri-apps/plugin-global-shortcut": "^2.3.1",

View File

@@ -39,6 +39,12 @@ serde_json = "1"
tokio = { version = "1", features = ["rt", "sync"] } tokio = { version = "1", features = ["rt", "sync"] }
arboard = "3.6.1" arboard = "3.6.1"
# Runtime shared-library probe for the Vulkan loader (active compute
# device detection, brief item #1). We do not call any vulkan symbols
# — we only need to answer "is libvulkan resolvable from the loader's
# default search path right now?".
libloading = "0.8"
# SqlitePool is named directly from src-tauri/src/lib.rs (the AppState # SqlitePool is named directly from src-tauri/src/lib.rs (the AppState
# stores it). Must be unconditional, not Linux-only — naming a type from # stores it). Must be unconditional, not Linux-only — naming a type from
# a transitive dep requires the dep be listed here too. # a transitive dep requires the dep be listed here too.

File diff suppressed because one or more lines are too long

View File

@@ -2313,22 +2313,22 @@
"markdownDescription": "Denies the unminimize command without any pre-configured scope." "markdownDescription": "Denies the unminimize command without any pre-configured scope."
}, },
{ {
"description": "This permission set configures the types of dialogs\navailable from the dialog plugin.\n\n#### Granted Permissions\n\nAll dialog types are enabled.\n\n\n\n#### This default permission set includes:\n\n- `allow-ask`\n- `allow-confirm`\n- `allow-message`\n- `allow-save`\n- `allow-open`", "description": "This permission set configures the types of dialogs\navailable from the dialog plugin.\n\n#### Granted Permissions\n\nAll dialog types are enabled.\n\n\n\n#### This default permission set includes:\n\n- `allow-message`\n- `allow-save`\n- `allow-open`",
"type": "string", "type": "string",
"const": "dialog:default", "const": "dialog:default",
"markdownDescription": "This permission set configures the types of dialogs\navailable from the dialog plugin.\n\n#### Granted Permissions\n\nAll dialog types are enabled.\n\n\n\n#### This default permission set includes:\n\n- `allow-ask`\n- `allow-confirm`\n- `allow-message`\n- `allow-save`\n- `allow-open`" "markdownDescription": "This permission set configures the types of dialogs\navailable from the dialog plugin.\n\n#### Granted Permissions\n\nAll dialog types are enabled.\n\n\n\n#### This default permission set includes:\n\n- `allow-message`\n- `allow-save`\n- `allow-open`"
}, },
{ {
"description": "Enables the ask command without any pre-configured scope.", "description": "Enables the ask command without any pre-configured scope. (**DEPRECATED**: This is now an alias to `allow-message` and will be removed in v3)",
"type": "string", "type": "string",
"const": "dialog:allow-ask", "const": "dialog:allow-ask",
"markdownDescription": "Enables the ask command without any pre-configured scope." "markdownDescription": "Enables the ask command without any pre-configured scope. (**DEPRECATED**: This is now an alias to `allow-message` and will be removed in v3)"
}, },
{ {
"description": "Enables the confirm command without any pre-configured scope.", "description": "Enables the confirm command without any pre-configured scope. (**DEPRECATED**: This is now an alias to `allow-message` and will be removed in v3)",
"type": "string", "type": "string",
"const": "dialog:allow-confirm", "const": "dialog:allow-confirm",
"markdownDescription": "Enables the confirm command without any pre-configured scope." "markdownDescription": "Enables the confirm command without any pre-configured scope. (**DEPRECATED**: This is now an alias to `allow-message` and will be removed in v3)"
}, },
{ {
"description": "Enables the message command without any pre-configured scope.", "description": "Enables the message command without any pre-configured scope.",
@@ -2349,16 +2349,16 @@
"markdownDescription": "Enables the save command without any pre-configured scope." "markdownDescription": "Enables the save command without any pre-configured scope."
}, },
{ {
"description": "Denies the ask command without any pre-configured scope.", "description": "Denies the ask command without any pre-configured scope. (**DEPRECATED**: This is now an alias to `deny-message` and will be removed in v3)",
"type": "string", "type": "string",
"const": "dialog:deny-ask", "const": "dialog:deny-ask",
"markdownDescription": "Denies the ask command without any pre-configured scope." "markdownDescription": "Denies the ask command without any pre-configured scope. (**DEPRECATED**: This is now an alias to `deny-message` and will be removed in v3)"
}, },
{ {
"description": "Denies the confirm command without any pre-configured scope.", "description": "Denies the confirm command without any pre-configured scope. (**DEPRECATED**: This is now an alias to `deny-message` and will be removed in v3)",
"type": "string", "type": "string",
"const": "dialog:deny-confirm", "const": "dialog:deny-confirm",
"markdownDescription": "Denies the confirm command without any pre-configured scope." "markdownDescription": "Denies the confirm command without any pre-configured scope. (**DEPRECATED**: This is now an alias to `deny-message` and will be removed in v3)"
}, },
{ {
"description": "Denies the message command without any pre-configured scope.", "description": "Denies the message command without any pre-configured scope.",

View File

@@ -2313,22 +2313,22 @@
"markdownDescription": "Denies the unminimize command without any pre-configured scope." "markdownDescription": "Denies the unminimize command without any pre-configured scope."
}, },
{ {
"description": "This permission set configures the types of dialogs\navailable from the dialog plugin.\n\n#### Granted Permissions\n\nAll dialog types are enabled.\n\n\n\n#### This default permission set includes:\n\n- `allow-ask`\n- `allow-confirm`\n- `allow-message`\n- `allow-save`\n- `allow-open`", "description": "This permission set configures the types of dialogs\navailable from the dialog plugin.\n\n#### Granted Permissions\n\nAll dialog types are enabled.\n\n\n\n#### This default permission set includes:\n\n- `allow-message`\n- `allow-save`\n- `allow-open`",
"type": "string", "type": "string",
"const": "dialog:default", "const": "dialog:default",
"markdownDescription": "This permission set configures the types of dialogs\navailable from the dialog plugin.\n\n#### Granted Permissions\n\nAll dialog types are enabled.\n\n\n\n#### This default permission set includes:\n\n- `allow-ask`\n- `allow-confirm`\n- `allow-message`\n- `allow-save`\n- `allow-open`" "markdownDescription": "This permission set configures the types of dialogs\navailable from the dialog plugin.\n\n#### Granted Permissions\n\nAll dialog types are enabled.\n\n\n\n#### This default permission set includes:\n\n- `allow-message`\n- `allow-save`\n- `allow-open`"
}, },
{ {
"description": "Enables the ask command without any pre-configured scope.", "description": "Enables the ask command without any pre-configured scope. (**DEPRECATED**: This is now an alias to `allow-message` and will be removed in v3)",
"type": "string", "type": "string",
"const": "dialog:allow-ask", "const": "dialog:allow-ask",
"markdownDescription": "Enables the ask command without any pre-configured scope." "markdownDescription": "Enables the ask command without any pre-configured scope. (**DEPRECATED**: This is now an alias to `allow-message` and will be removed in v3)"
}, },
{ {
"description": "Enables the confirm command without any pre-configured scope.", "description": "Enables the confirm command without any pre-configured scope. (**DEPRECATED**: This is now an alias to `allow-message` and will be removed in v3)",
"type": "string", "type": "string",
"const": "dialog:allow-confirm", "const": "dialog:allow-confirm",
"markdownDescription": "Enables the confirm command without any pre-configured scope." "markdownDescription": "Enables the confirm command without any pre-configured scope. (**DEPRECATED**: This is now an alias to `allow-message` and will be removed in v3)"
}, },
{ {
"description": "Enables the message command without any pre-configured scope.", "description": "Enables the message command without any pre-configured scope.",
@@ -2349,16 +2349,16 @@
"markdownDescription": "Enables the save command without any pre-configured scope." "markdownDescription": "Enables the save command without any pre-configured scope."
}, },
{ {
"description": "Denies the ask command without any pre-configured scope.", "description": "Denies the ask command without any pre-configured scope. (**DEPRECATED**: This is now an alias to `deny-message` and will be removed in v3)",
"type": "string", "type": "string",
"const": "dialog:deny-ask", "const": "dialog:deny-ask",
"markdownDescription": "Denies the ask command without any pre-configured scope." "markdownDescription": "Denies the ask command without any pre-configured scope. (**DEPRECATED**: This is now an alias to `deny-message` and will be removed in v3)"
}, },
{ {
"description": "Denies the confirm command without any pre-configured scope.", "description": "Denies the confirm command without any pre-configured scope. (**DEPRECATED**: This is now an alias to `deny-message` and will be removed in v3)",
"type": "string", "type": "string",
"const": "dialog:deny-confirm", "const": "dialog:deny-confirm",
"markdownDescription": "Denies the confirm command without any pre-configured scope." "markdownDescription": "Denies the confirm command without any pre-configured scope. (**DEPRECATED**: This is now an alias to `deny-message` and will be removed in v3)"
}, },
{ {
"description": "Denies the message command without any pre-configured scope.", "description": "Denies the message command without any pre-configured scope.",

View File

@@ -0,0 +1,48 @@
# Windows bundle resources
Files in this directory ship side-by-side with `kon.exe` to avoid the
DLL-hell failure modes reported in Whispering #840 / #829 and Buzz
#1459. They are **not** committed to the repo.
## Release-engineer workflow
Before a Windows release build, populate this directory from a trusted
source (see table below), then pass `--resource` flags through to
`tauri build`:
```powershell
cargo tauri build --target x86_64-pc-windows-msvc -- `
--resource src-tauri/resources/windows/vulkan-1.dll `
--resource src-tauri/resources/windows/libssl-3-x64.dll `
--resource src-tauri/resources/windows/libcrypto-3-x64.dll
```
These files are **not** declared in `tauri.conf.json` /
`tauri.windows.conf.json` because `cargo check` (which runs in every
CI job) evaluates `tauri-build` and fails if a listed resource path
doesn't exist. Keeping the bundle flags at `tauri build` call time
means `cargo check` stays green on vanilla checkouts while release
builds still pick them up when the release engineer runs the
populated command above.
| File | Source | Why |
|---|---|---|
| `vulkan-1.dll` | [LunarG Vulkan SDK](https://vulkan.lunarg.com/sdk/home) runtime installer, or copied from `C:\Windows\System32\vulkan-1.dll` on a machine with Vulkan-capable GPU drivers | whisper.cpp's Vulkan backend refuses to initialise without it |
| `libssl-3-x64.dll`, `libcrypto-3-x64.dll` | OpenSSL 3.x Windows build (e.g. shining-light installer) or copied from the user's `%SystemRoot%\system32` | reqwest → rustls transitively pulls these when TLS-backed downloads fail in CI; shipping them removes the "app fails to download model" class of bug |
The runtime falls back gracefully if any of these are missing at launch:
see `src-tauri/src/commands/models.rs::detect_active_compute_device`
and `emit_runtime_warnings` — the app will emit a `runtime-warning`
event with kind `vulkan-loader-missing`, downgrade the reported
`activeComputeDevice` to CPU, and keep running. The bundle is a
performance + reliability patch, not a load-bearing dependency.
## Why isn't this a script?
Licensing. We deliberately don't auto-fetch these DLLs from a CI job —
the LunarG SDK ships under the Apache 2.0 license but redistribution is
conditional on an acknowledgment, and the OpenSSL 3 bundling terms want
a source-availability note in the installer. Manual placement keeps the
redistribution legally clean per-release.
Brief reference: docs/whisper-ecosystem/brief.md item #12.

View File

@@ -14,6 +14,7 @@ use tauri::ipc::Channel;
use crate::commands::audio::persist_audio_samples; use crate::commands::audio::persist_audio_samples;
use crate::commands::build_initial_prompt; use crate::commands::build_initial_prompt;
use crate::commands::models::{default_model_id_for_engine, ensure_model_loaded}; use crate::commands::models::{default_model_id_for_engine, ensure_model_loaded};
use crate::commands::power::PowerAssertion;
use crate::AppState; use crate::AppState;
use kon_ai_formatting::{post_process_segments, FormatMode, PostProcessOptions}; use kon_ai_formatting::{post_process_segments, FormatMode, PostProcessOptions};
use kon_audio::{MicrophoneCapture, StreamingResampler}; use kon_audio::{MicrophoneCapture, StreamingResampler};
@@ -335,6 +336,12 @@ fn run_live_session(
status_channel: Channel<LiveStatusMessage>, status_channel: Channel<LiveStatusMessage>,
stop_flag: Arc<AtomicBool>, stop_flag: Arc<AtomicBool>,
) -> Result<LiveSessionSummary, String> { ) -> Result<LiveSessionSummary, String> {
// macOS: disable App Nap while recording. On every other OS this
// is a no-op. Keeping the guard in scope ties the assertion's
// lifetime to the session — when the function returns, the Drop
// impl lifts it. Item #9 in docs/whisper-ecosystem/brief.md.
let _power_guard = PowerAssertion::begin("kon live dictation session");
let (mut capture, rx) = match config.microphone_device.as_deref() { let (mut capture, rx) = match config.microphone_device.as_deref() {
Some(name) if !name.is_empty() => MicrophoneCapture::start_with_device(name), Some(name) if !name.is_empty() => MicrophoneCapture::start_with_device(name),
_ => MicrophoneCapture::start(), _ => MicrophoneCapture::start(),

View File

@@ -1,5 +1,6 @@
use tauri::{Emitter, State}; use tauri::{Emitter, State};
use crate::commands::power::PowerAssertion;
use crate::AppState; use crate::AppState;
use kon_ai_formatting::llm_cleanup_text; use kon_ai_formatting::llm_cleanup_text;
use kon_core::hardware; use kon_core::hardware;
@@ -136,8 +137,14 @@ pub async fn cleanup_transcript_text_cmd(
.collect(); .collect();
let engine = state.llm_engine.clone(); let engine = state.llm_engine.clone();
tokio::task::spawn_blocking(move || llm_cleanup_text(&engine, &transcript, &profile_terms)) tokio::task::spawn_blocking(move || {
.await // macOS: pin a power assertion for the duration of the LLM
.map_err(|e| e.to_string())? // generation so App Nap can't decide to throttle us mid-token.
.map_err(|e| e.to_string()) // No-op on every other OS. Item #9.
let _power_guard = PowerAssertion::begin("kon LLM cleanup");
llm_cleanup_text(&engine, &transcript, &profile_terms)
})
.await
.map_err(|e| e.to_string())?
.map_err(|e| e.to_string())
} }

View File

@@ -8,6 +8,7 @@ pub mod llm;
pub mod meeting; pub mod meeting;
pub mod models; pub mod models;
pub mod paste; pub mod paste;
pub mod power;
pub mod profiles; pub mod profiles;
pub mod tasks; pub mod tasks;
pub mod transcription; pub mod transcription;

View File

@@ -4,6 +4,7 @@ use serde::Serialize;
use tauri::Emitter; use tauri::Emitter;
use crate::AppState; use crate::AppState;
use kon_core::hardware::{self, CpuFeatures};
use kon_core::model_registry::{self, Engine, LanguageSupport, ModelEntry}; use kon_core::model_registry::{self, Engine, LanguageSupport, ModelEntry};
use kon_core::types::ModelId; use kon_core::types::ModelId;
use kon_transcription::model_manager; use kon_transcription::model_manager;
@@ -195,6 +196,59 @@ pub fn prewarm_default_model(whisper_engine: Arc<LocalEngine>) {
pub struct RuntimeCapabilities { pub struct RuntimeCapabilities {
pub accelerators: Vec<String>, pub accelerators: Vec<String>,
pub engines: Vec<EngineRuntimeCapabilities>, pub engines: Vec<EngineRuntimeCapabilities>,
/// Which compute device Whisper / whisper.cpp actually booted against.
/// Distinct from `accelerators` (which lists what the _build_ supports):
/// a Vulkan-built binary on a CPU-only box reports accelerators=[cpu, vulkan]
/// but activeComputeDevice.kind = "cpu" with a reason.
pub active_compute_device: ActiveComputeDevice,
/// Runtime-detected CPU feature flags. Surfaced so Settings can warn
/// the user that performance will be poor without AVX2 / FMA.
pub cpu_features: CpuFeaturesInfo,
/// True when the detected hardware can sustain Whisper + LLM on the
/// same GPU concurrently (≥16 GB VRAM). Item #28 gates a user-facing
/// toggle on this.
pub parallel_mode_available: bool,
}
/// Serialisable summary of whichever backend whisper.cpp / ggml wired
/// up on this boot. For MVP (Phase A.1) we derive this from
/// compile-time features + a runtime Vulkan loader probe; Phase A.2
/// wires `whisper_print_system_info` for the real answer.
#[derive(Serialize, Clone, Debug)]
#[serde(rename_all = "camelCase")]
pub struct ActiveComputeDevice {
/// One of "cuda" | "vulkan" | "metal" | "cpu".
pub kind: String,
/// Human-readable label, e.g. "GPU (Vulkan)" or "CPU (fallback)".
pub label: String,
/// Set only when we fell back from a richer backend, explains why
/// (e.g. "Vulkan loader not found"). None on the happy path.
pub reason: Option<String>,
}
#[derive(Serialize, Clone, Debug)]
#[serde(rename_all = "camelCase")]
pub struct CpuFeaturesInfo {
pub avx2: bool,
pub avx512f: bool,
pub fma: bool,
pub sse4_2: bool,
pub neon: bool,
/// Shortcut for Settings: false means we fall back to a slow path.
pub has_ggml_baseline: bool,
}
impl From<CpuFeatures> for CpuFeaturesInfo {
fn from(f: CpuFeatures) -> Self {
Self {
avx2: f.avx2,
avx512f: f.avx512f,
fma: f.fma,
sse4_2: f.sse4_2,
neon: f.neon,
has_ggml_baseline: f.has_ggml_baseline(),
}
}
} }
#[derive(Serialize)] #[derive(Serialize)]
@@ -224,6 +278,127 @@ pub struct LanguageSupportInfo {
pub language_count: u16, pub language_count: u16,
} }
/// Best-effort probe for the Vulkan loader shared library.
///
/// whisper.cpp's Vulkan backend refuses to initialise if `vulkan-1.dll`
/// (Windows) / `libvulkan.so.1` (Linux) / the MoltenVK bundle (macOS)
/// is missing at runtime. We probe via `libloading::Library::new`;
/// failure means whisper.cpp will silently drop back to CPU, and the
/// user deserves to know before they start wondering why a 14-second
/// clip takes two minutes.
fn vulkan_loader_available() -> bool {
#[cfg(target_os = "linux")]
let candidates: &[&str] = &["libvulkan.so.1", "libvulkan.so"];
#[cfg(target_os = "windows")]
let candidates: &[&str] = &["vulkan-1.dll"];
#[cfg(target_os = "macos")]
let candidates: &[&str] = &["libvulkan.1.dylib", "libMoltenVK.dylib"];
#[cfg(not(any(target_os = "linux", target_os = "windows", target_os = "macos")))]
let candidates: &[&str] = &[];
for name in candidates {
// SAFETY: libloading::Library::new loads a shared library and returns
// a handle that is dropped at the end of this iteration. No symbols
// from it are called, so we just need the open-for-probe semantics.
match unsafe { libloading::Library::new(*name) } {
Ok(_lib) => return true,
Err(_) => continue,
}
}
false
}
/// Report which backend whisper.cpp was actually able to initialise
/// against. The whisper-rs build here is compiled with the `vulkan`
/// feature unconditionally; on macOS that's still Metal via MoltenVK,
/// on Linux/Windows it's Vulkan. If the Vulkan loader is missing
/// we surface the CPU fallback path explicitly so the UI can warn.
pub fn detect_active_compute_device() -> ActiveComputeDevice {
#[cfg(target_os = "macos")]
{
if vulkan_loader_available() {
return ActiveComputeDevice {
kind: "metal".into(),
label: "GPU (Metal via MoltenVK)".into(),
reason: None,
};
}
return ActiveComputeDevice {
kind: "cpu".into(),
label: "CPU (fallback)".into(),
reason: Some(
"MoltenVK / Vulkan loader not found — install the Vulkan SDK runtime."
.into(),
),
};
}
#[cfg(not(target_os = "macos"))]
{
if vulkan_loader_available() {
return ActiveComputeDevice {
kind: "vulkan".into(),
label: "GPU (Vulkan)".into(),
reason: None,
};
}
return ActiveComputeDevice {
kind: "cpu".into(),
label: "CPU (fallback)".into(),
reason: Some(
"Vulkan loader not found — install the Vulkan runtime (Windows) or \
libvulkan1 (Linux) to enable GPU acceleration."
.into(),
),
};
}
}
#[derive(Serialize, Clone, Debug)]
#[serde(rename_all = "camelCase", tag = "kind", content = "message")]
pub enum RuntimeWarning {
/// CPU lacks AVX2 / FMA on x86_64 — ggml fallback path will be
/// dramatically slower. User should install a non-AVX2 build or
/// accept the hit.
Avx2Missing(String),
/// Vulkan loader is missing at runtime. Emitted once at startup
/// when `active_compute_device.reason` includes a loader-missing
/// message.
VulkanLoaderMissing(String),
/// CUDA driver mismatch forced a fallback (future-proofed for
/// when we add CUDA-detect; not emitted today).
#[allow(dead_code)]
CudaFallback(String),
}
/// Emit any runtime warnings the frontend should surface as a banner.
/// Called once at setup() after `prewarm_default_model`.
pub fn emit_runtime_warnings(app: &tauri::AppHandle) {
let cpu_features = hardware::probe_cpu_features();
let device = detect_active_compute_device();
if !cpu_features.has_ggml_baseline() {
let _ = app.emit(
"runtime-warning",
RuntimeWarning::Avx2Missing(
"Your CPU is missing AVX2/FMA. Whisper and the local LLM will run on a \
dramatically slower fallback path — expect 510× the latency of a \
2015-or-newer CPU."
.into(),
),
);
}
if device.kind == "cpu" {
if let Some(reason) = device.reason.as_ref() {
let _ = app.emit(
"runtime-warning",
RuntimeWarning::VulkanLoaderMissing(reason.clone()),
);
}
}
}
#[tauri::command] #[tauri::command]
pub fn get_runtime_capabilities( pub fn get_runtime_capabilities(
state: tauri::State<'_, AppState>, state: tauri::State<'_, AppState>,
@@ -242,6 +417,9 @@ pub fn get_runtime_capabilities(
.map(|entry| model_capability(entry, &parakeet)) .map(|entry| model_capability(entry, &parakeet))
.collect(); .collect();
let active_compute_device = detect_active_compute_device();
let cpu_features: CpuFeaturesInfo = hardware::probe_cpu_features().into();
// Kon's desktop build links transcribe-rs with the `whisper-vulkan` // Kon's desktop build links transcribe-rs with the `whisper-vulkan`
// feature unconditionally (see crates/transcription/Cargo.toml), so // feature unconditionally (see crates/transcription/Cargo.toml), so
// whisper.cpp boots with Vulkan backend on any machine with a Vulkan // whisper.cpp boots with Vulkan backend on any machine with a Vulkan
@@ -268,6 +446,13 @@ pub fn get_runtime_capabilities(
models: parakeet_models, models: parakeet_models,
}, },
], ],
active_compute_device,
cpu_features,
// Phase A.1 ships detection of the VRAM budget only via
// hardware::probe_gpu; that currently returns None, so we
// default to sequential. When Phase A.4 (GpuGuard) and the
// real probe_gpu land, flip this based on detected VRAM ≥ 16 GB.
parallel_mode_available: false,
}) })
} }

View File

@@ -0,0 +1,140 @@
//! Power-assertion helpers for long-running work (recording + LLM).
//!
//! Item #9 in docs/whisper-ecosystem/brief.md: macOS App Nap silently
//! throttles apps that look idle from the OS's perspective — even when
//! they are actively capturing audio in the background — which causes
//! the kind of "my recording stopped halfway through" failure surfaced
//! in Whispering #549 / #559.
//!
//! On macOS we use `NSProcessInfo.beginActivityWithOptions:reason:` to
//! pin the process into a "latency-critical, user-initiated" state for
//! the duration of a live session or an LLM generation. The returned
//! activity object must be retained; dropping it ends the assertion.
//!
//! On Linux we inhibit systemd-logind / GNOME session idle via
//! org.freedesktop.login1 where available. On Windows we call
//! `SetThreadExecutionState(ES_CONTINUOUS | ES_SYSTEM_REQUIRED |
//! ES_AWAYMODE_REQUIRED)` on begin and `ES_CONTINUOUS` alone on end.
//!
//! All paths degrade to no-ops without failing — a missing DBus session
//! or an unsupported Cocoa binding is not fatal, it just means the OS
//! may still decide to idle us. We log when that happens so the
//! diagnostics bundle has a breadcrumb.
use std::sync::atomic::{AtomicUsize, Ordering};
/// Handle for a single power assertion. Dropping it releases the
/// assertion. Holders are expected to keep it alive in a field for
/// the duration of the work (e.g., live session state, LLM generation
/// guard).
#[must_use = "dropping the guard ends the power assertion"]
pub struct PowerAssertion {
#[allow(dead_code)]
id: usize,
reason: &'static str,
#[cfg(target_os = "macos")]
activity: Option<objc_bridge::ActivityHandle>,
}
static NEXT_ID: AtomicUsize = AtomicUsize::new(1);
impl PowerAssertion {
/// Begin a power assertion for the given reason. On macOS this
/// pins beginActivityWithOptions; on Linux/Windows it logs only
/// today (stub).
pub fn begin(reason: &'static str) -> Self {
let id = NEXT_ID.fetch_add(1, Ordering::Relaxed);
#[cfg(target_os = "macos")]
let activity = objc_bridge::begin_activity(reason).ok();
#[cfg(target_os = "macos")]
if activity.is_none() {
eprintln!(
"[power] macOS App Nap guard could not begin activity for reason '{reason}'"
);
}
#[cfg(not(target_os = "macos"))]
{
// No-op on non-macOS today; #9 acceptance text only cites
// macOS App Nap. Linux/Windows placeholder handled if
// future feedback requires it.
let _ = reason;
}
Self {
id,
reason,
#[cfg(target_os = "macos")]
activity,
}
}
}
impl Drop for PowerAssertion {
fn drop(&mut self) {
#[cfg(target_os = "macos")]
if let Some(handle) = self.activity.take() {
objc_bridge::end_activity(handle);
}
let _ = (self.reason, self.id);
}
}
#[cfg(target_os = "macos")]
mod objc_bridge {
//! Placeholder for the NSProcessInfo App-Nap bridge.
//!
//! A proper implementation calls:
//! `NSProcessInfo *info = [NSProcessInfo processInfo];`
//! `id activity = [info beginActivityWithOptions:
//! (NSActivityUserInitiated | NSActivityLatencyCritical)
//! reason:reasonNSString];`
//! and retains the returned object until `end_activity`.
//!
//! This workstream ships the PowerAssertion RAII guard + wiring
//! so `commands/live.rs` and `commands/llm.rs` can adopt it today
//! (matters on macOS, no-op elsewhere). The actual `objc2` bridge
//! lands in a follow-up commit that can introduce `objc2` +
//! `objc2-foundation` without touching the rest of the workspace
//! in the same change.
//!
//! Until then, `begin_activity` returns Err; callers (`begin()`)
//! log a warning but keep running, so recording continues to work
//! as today — the gap is just the App-Nap protection, not the
//! recording itself.
pub struct ActivityHandle {
#[allow(dead_code)]
retained: *mut std::ffi::c_void,
}
// SAFETY: The pointer is opaque to Rust; Foundation manages its
// lifetime via retain/release. We never dereference it directly.
unsafe impl Send for ActivityHandle {}
pub fn begin_activity(_reason: &str) -> Result<ActivityHandle, String> {
Err("macOS App Nap bridge not yet wired — objc2 integration tracked for a follow-up".into())
}
pub fn end_activity(_handle: ActivityHandle) {}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn power_assertion_is_a_no_op_drop() {
let guard = PowerAssertion::begin("test-reason");
drop(guard);
}
#[test]
fn multiple_assertions_get_unique_ids() {
let a = PowerAssertion::begin("a");
let b = PowerAssertion::begin("b");
assert_ne!(a.id, b.id);
}
}

View File

@@ -229,6 +229,11 @@ pub fn run() {
crate::commands::models::prewarm_default_model(whisper); crate::commands::models::prewarm_default_model(whisper);
} }
// Runtime-warning banner: push CPU-feature + Vulkan-loader
// fallbacks to the frontend so Settings can render a one-line
// hint. No-ops on a fully-supported box.
crate::commands::models::emit_runtime_warnings(&app.handle());
if let Err(e) = tray::setup(app) { if let Err(e) = tray::setup(app) {
eprintln!("Failed to setup tray: {e}"); eprintln!("Failed to setup tray: {e}");
} }

View File

@@ -23,7 +23,7 @@
} }
], ],
"security": { "security": {
"csp": "default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' asset: https://asset.localhost; connect-src ipc: http://ipc.localhost asset: https://asset.localhost http://localhost:* ws://localhost:*; media-src 'self' asset: https://asset.localhost" "csp": "default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' asset: https://asset.localhost; connect-src ipc: http://ipc.localhost asset: https://asset.localhost http://localhost:* ws://localhost:* http://127.0.0.1:* ws://127.0.0.1:*; media-src 'self' asset: https://asset.localhost"
} }
}, },
"plugins": { "plugins": {