Lumotia/docs/whisper-ecosystem/kon-context.md

Kon — Engineering Context for Cursor Agents

Canonical project context for Workstream A (Codex) and Workstream B (Opus). If you are a cloud-based AI agent working on Kon via Cursor, read this before reading brief.md — it tells you what is already shipped, what is intentionally deferred, and the ideology rules you cannot override.

---

What Kon is

Kon is a local-first, cognitive-load-aware dictation + task-capture desktop app. Stack: Tauri 2 + Rust workspace + Svelte 5 frontend. Current primary target is Linux (KDE Plasma Wayland); macOS and Windows are in scope. All transcription and LLM inference run locally — no cloud call is ever made implicitly.

Key paths:

• crates/ — Rust workspace (core, audio, transcription, llm, ai-formatting, storage, hotkey, cloud-providers, mcp)
• src-tauri/ — Tauri application (binary + commands)
• src/ — SvelteKit frontend (routes, lib/pages, lib/components, lib/stores)
• docs/whisper-ecosystem/brief.md — cross-repo research informing this workstream pair
• HANDOVER.md, HANDOVER-2026-04-18.md, HANDOVER-2026-04-17.md — session handovers you should skim for recent decisions

---

Non-negotiable design rules

These are load-bearing. If a task seems to require violating one, stop and flag it; do not "just do it."

1. LLM / agent scope is narrow

The in-app LLM does exactly two things: post-transcription formatting cleanup and task decomposition / extraction. Nothing else.

Do not propose or build:

• Wake-word detection or always-listening agent flows
• Conversational / chat UI for the LLM
• Multi-provider cloud fan-out (Bedrock, Vertex, Azure, Groq, etc.)
• Agent "skills" registries or tool-use hooks driven by spoken intent

Do keep LLM touch points focused on:

• Formatting the transcript (done by cleanup_transcript_text_cmd, prompt in crates/ai-formatting/src/llm_client.rs)
• Decomposing a task into 3–7 physical micro-steps (decompose_and_store)
• Extracting action items from a transcript (extract_tasks_from_transcript_cmd)
• Injecting domain vocabulary as Whisper initial_prompt and into the cleanup prompt

Cloud provider support exists (crates/cloud-providers) but is empty. When it grows, ceiling is an OpenAI-compatible endpoint + Anthropic. Do not add more.

2. No second notes surface

Kon is not a notes app. The transcript surface is History + YAML-frontmatter markdown export to the user's existing notes tool (Obsidian). Transcript editing (fix recognition errors in the viewer window) is allowed. Tagging, starring, searching transcripts is allowed. Anything that starts to look like "edit long-form prose" or "sync notes to another device/cloud" is out of scope — stop and reconsider.

Do not propose or build:

• A Tiptap / ProseMirror / rich-text editor
• A dedicated "notes" route
• Cloud note-sync

3. Meeting auto-capture: opt-in, single-signal, passive

If meeting detection is touched, it must be off by default, use only process-list polling as a signal, and surface a non-modal reminder — never start recording autonomously. This is already shipped (src-tauri/src/commands/meeting.rs, crates/core/src/process_watch.rs); do not add mic-activity heuristics or calendar integration.

4. Raw transcript is always recoverable

The user's words as Whisper heard them are sacrosanct. LLM cleanup can run by default, but the raw transcript must always be available for revert (task #17 in brief.md). Do not rewrite history in the paste buffer; do not discard raw segments after cleanup.

5. Local-first

No feature may assume cloud availability. Offline install is the baseline; cloud is optional polish.

6. Low cognitive load

Kon is deliberately ADHD-aware. A new setting must earn its mental real estate. A new flow must reduce not add steps. When in doubt, pick the option that requires fewer user decisions.

---

Architectural decisions (do not re-litigate)

Whisper runtime: whisper-rs + Vulkan

Kon uses whisper-rs 0.16 with the vulkan feature (crates/transcription/Cargo.toml), wrapping whisper.cpp. Do NOT swap to faster-whisper / WhisperX / any PyTorch-based fork. Both drag a Python runtime into the Tauri binary, which is a far larger tax than any speedup justifies. Runtime-side improvements go into whisper-rs / whisper.cpp. Model-side improvements (Distil-Whisper GGUF, Parakeet-as-English-default) are already shipped.

The brief.md references faster-whisper as a pattern source — treat it as one. Do not adopt it as a dependency.

LLM runtime: llama-cpp-2 with Qwen3 tiers

crates/llm/ owns the LLM runtime. Three tiers (Qwen3 1.7B, Qwen3 4B-Instruct-2507, Qwen3 14B) selected via hardware probe. Resumable HTTP downloads with SHA-256 verification. Do not add alternative runtimes (candle, mistral.rs, etc.) without strong evidence — the current stack is working.

GPU compatibility: Vulkan everywhere

Both whisper-rs and llama-cpp-2 link the Vulkan feature. This intentionally sidesteps CUDA version hell on Windows and works across NVIDIA / AMD / Intel / macOS (via MoltenVK). Do not introduce CUDA-specific code paths.

Hotkey backend

• Linux (X11 and Wayland): evdev via crates/hotkey/src/linux.rs. Requires the user in the input group; error messaging already surfaces this.
• macOS / Windows: Tauri's tauri-plugin-global-shortcut.
• Cross-platform logic lives in src/routes/+layout.svelte.

ggml dedup (interim)

Both llama-cpp-sys-2 and whisper-rs-sys statically link their own ggml. On Linux, src-tauri/build.rs emits -Wl,--allow-multiple-definition as an interim workaround. Do not attempt to fix this in either workstream — a proper system-ggml shared-lib setup is its own workstream and out of scope here.

Window management

Tauri 2 windows with tauri-plugin-window-state for position/size persistence. Preview overlay (/preview route) is always-on-top, skip_taskbar, visible_on_all_workspaces, and has WindowTypeHint::Utility set via GTK on Linux. Focus-gated: only opens when the main window is unfocused at record-start.

---

What is already shipped on main (DO NOT re-implement)

The phase3-llm-runtime branch just merged — 16 commits, summary follows. When in doubt, grep before implementing.

Capability	File pointers
Local LLM runtime (load / unload / cleanup / extract_tasks / decompose_task)	crates/llm/src/lib.rs, crates/llm/src/model_manager.rs
LLM commands (tier recommend / download / load / unload / delete / status / cleanup / extract / decompose)	src-tauri/src/commands/llm.rs, src-tauri/src/commands/tasks.rs
Whisper initial_prompt built from caller prompt + profile prompt + profile_terms	src-tauri/src/commands/mod.rs::build_initial_prompt
Distil-Whisper Small + Large v3 entries	crates/core/src/model_registry.rs
Parakeet-as-default-for-English (scored in recommendation)	crates/core/src/recommendation.rs
Platform paste matrix (wtype / xdotool / ydotool / osascript / SendKeys) with Wayland focus-race mitigation	src-tauri/src/commands/paste.rs
i18n scaffolding (svelte-i18n, en/es/de, language selector)	src/lib/i18n/, src/lib/pages/SettingsPage.svelte
MCP stdio server (read-only tools: list/search/get transcripts, list tasks)	crates/mcp/
Meeting auto-capture (opt-in, process-list poll, edge-triggered toast)	src-tauri/src/commands/meeting.rs
FTS5-backed transcript search	crates/storage/src/database.rs::search_transcripts
Transcription preview overlay (listening → live → cleanup → final → auto-hide)	src/routes/preview/, src-tauri/src/commands/windows.rs::open_preview_window
Bulk profile-terms import	src/lib/pages/SettingsPage.svelte::addBulkVocabTerms
Per-window size + position persistence	tauri-plugin-window-state registered in src-tauri/src/lib.rs

This means item #18 in brief.md (initial prompt / custom-vocab priming) is already shipped — verify and skip, don't re-implement.

Partial coverage worth extending (not re-writing):

• Hallucination blocklist (#22) — partial via ai-formatting::rule_based::is_hallucination. Extend with avg_logprob / no_speech_prob gate, don't replace the regex list.
• Engine abstraction (#13) — partial via LocalEngine wrapping both whisper and parakeet paths. Unify rather than greenfield.
• Warm-up (#23) — model is pre-warmed via prewarm_default_model. Silent-WAV inference pass is the missing piece.
• Plain-text LLM input (#29) — already joined as text before cleanup in ai-formatting::pipeline. Verify timestamps are stripped; likely a one-line confirmation.

---

Intentionally deferred (do NOT work on these)

Voice calibration roadmap (three tiers: mic-levels → passage → continuous)

A per-user speech-gate + initial_prompt priming flow. Discussed and scoped, but not in either current workstream. The hardcoded speech-gate constants in src-tauri/src/commands/live.rs lines 29–34 stay as-is for now.

OpenWhispr audit items already decided-skipped

• AT-SPI2 terminal detection (Ctrl+Shift+V variant) — low ROI; most modern terminals accept Ctrl+V.
• Multi-language auto-detect + retry — brief item not aligned with either workstream; deferred until i18n has real multilingual users.
• Dictation panel visibility modes (Always / When Transcribing / Hidden) — current auto-gated behaviour covers it.
• Mic-button visual feedback polish — cosmetic; VisualTimer + UnicodeSpinner already exist.

Cloud-endpoint contract test

Pinned for when kon-cloud-providers actually grows a provider. Not actionable yet.

ggml system-lib dedup

Pinned as its own future workstream.

Speaker diarization

Deferred. If ever built, use sherpa-onnx (already linked for Parakeet).

Dragon-style "100-passage training"

Never. Whisper has no speaker adaptation; fine-tuning is out of Kon's shape.

---

Guard-rails by workstream

Workstream A (Codex) — Systems + streaming correctness

Your task list: items #1, #2, #6, #7, #8, #9, #12, #13, #19, #21, #22, #23, #24, #25, #26, #28, #29 from brief.md. Item #18 is already shipped — verify and skip.

You own:

• crates/transcription/**
• crates/audio/**
• crates/ai-formatting/src/{pipeline,rule_based}.rs
• src-tauri/src/commands/{models,transcription,live,audio}.rs
• src-tauri/tauri.conf.json (capabilities)
• src-tauri/build.rs (CPU feature detection — but do not touch the ggml link-arg)

You do NOT touch:

• Any src/** file (frontend)
• crates/ai-formatting/src/llm_client.rs
• src-tauri/src/commands/{paste,clipboard,hotkey,llm}.rs — unless the change is backend-only and flagged in your workstream-A.md as a shared-contract surface for Opus to consume
• The ggml link-arg in src-tauri/build.rs

Contract surfaces to design up front so Opus can consume them:

• get_active_compute_device command — returns { engine, backend, device_name, fallback_reason? }
• list_gpus command — enumerates GPUs for the Settings dropdown
• Streaming cleanup variant — cleanup_transcript_stream emitting kon:llm-token events, abortable via cancel_llm

Document these in docs/whisper-ecosystem/workstream-A.md before coding so Opus can stub frontends against the agreed shape.

Workstream B (Opus) — UX + formatting layer

Your task list: items #3, #4, #5, #10, #11, #14, #15, #16, #17, #20, #27, #30, #31 from brief.md.

You own:

• src/** (all frontend)
• src-tauri/src/commands/{paste,clipboard,hotkey,llm}.rs — enhance, don't replace
• crates/ai-formatting/src/llm_client.rs — only the CLEANUP_PROMPT constant (task #16). No other code in that file.

You do NOT touch:

• crates/transcription/**
• crates/audio/**
• crates/ai-formatting/src/{pipeline,rule_based}.rs
• src-tauri/src/commands/{models,transcription,live,audio}.rs

Ideology reminders that matter most for your scope:

• #17 raw-revert is where rule 4 ("raw transcript always recoverable") becomes concrete. Five-second ⌘/Ctrl+Z window, replaces LLM output with raw.
• #16 prompt framing — the "translator from spoken to written form, not an editor trying to improve the content" framing is load-bearing for rule 1. Do not drift from it; add a regression test that asserts the framing survives refactors.
• #15 named prompt presets — presets extend profile.initial_prompt, they do not replace it. Stay compatible with the existing per-profile vocabulary flow.
• #27 LLM test-connection — error classification is the point. "Ollama not installed" / "port blocked" / "model not pulled" / "scope error" — never a raw URL error to the user.

Frontend contract for tasks that need Codex's backend:

• #14 (GPU enum) needs list_gpus from Codex
• #30 (streaming LLM) needs cleanup_transcript_stream from Codex
• #1's chip UI (not in your list but adjacent) needs get_active_compute_device

If Codex hasn't shipped these yet, stub the frontend behind a feature flag with a clear TODO and ship the rest of your work — do not block.

---

Required reading, in order

1. This document (you're in it).
2. docs/whisper-ecosystem/brief.md — the 31-item atomic task backlog is your source of truth.
3. HANDOVER.md — latest session handover, Linux/Wayland specifics.
4. The specific Kon file pointers referenced under your workstream's "You own" list.

Then write your docs/whisper-ecosystem/workstream-{A,B}.md execution plan (sequence, dependencies, contract shapes) and commit it before writing any non-doc code.