diff --git a/README.md b/README.md new file mode 100644 index 0000000..ff7bfdd --- /dev/null +++ b/README.md @@ -0,0 +1,377 @@ +# Kon + +*Think out loud. Keep working.* + +Kon is a local-first, cognitive-load-aware dictation and task-capture desktop app. Every transcription, LLM cleanup, and task extraction runs on the user's machine. No telemetry, no analytics, no cloud dependency. The app is designed around a single observation: people who think in bursts lose ideas faster than they can type, and the tool's job is to get out of the way. + +--- + +## Status + +**Pre-alpha.** Actively dogfooded on Linux (KDE Plasma 6 on Wayland). macOS and Windows targets are in scope and exercised by CI, but not yet beta-ready. One primary user; open source-intent with licence TBD before public beta. + +- Current `main`: see commit log +- 136 automated lib tests across 10 crates, all passing +- Cross-platform CI (Linux / macOS / Windows) via GitHub Actions + +--- + +## Design principles (non-negotiable) + +1. **Local-first is the floor, not a feature.** No voice, transcript, or task ever leaves the user's machine unless they explicitly send it. No telemetry. +2. **Cognitive load is the limiting resource.** Every new setting must earn its mental real estate. Every interaction should reduce, not add, decisions. +3. **Composable, not monolithic.** Kon is a dictation primitive: via MCP, CLI, and filesystem export, it slots into whatever workflow the user already has (Obsidian, Claude Desktop, Cline, any text field). +4. **LLM scope is narrow.** The in-app LLM does transcription cleanup and task extraction. It is not a wake-word agent, not a chat UI, not a multi-provider cloud fan-out. +5. **Raw transcript is always recoverable.** Cleanup is additive, never destructive. The user can always see and revert to what Whisper heard. + +These are enforced in the codebase (where practical) and in the docs under [`docs/whisper-ecosystem/kon-context.md`](docs/whisper-ecosystem/kon-context.md). + +--- + +## What Kon does today + +### Speech-to-text +- Vulkan-accelerated local **Whisper** inference via [whisper-rs](https://github.com/tazz4843/whisper-rs) 0.16 + whisper.cpp. Works on NVIDIA, AMD, Intel Arc, Apple (via MoltenVK), and integrated graphics. +- Vulkan / CUDA-accelerated **Parakeet** inference via sherpa-onnx (NVIDIA's English-only model; lower latency than Whisper-Large on English). +- **Six Whisper variants** shipped: Tiny, Base, Small, Distil-Small, Medium, Distil-Large v3. +- **Parakeet-as-default for English** when hardware supports it; first-run hardware probe picks the fastest-accurate pair. +- **Resumable downloads with SHA-256 verification**; retains audio if transcription fails. +- **Per-profile custom vocabulary** fed to Whisper as `initial_prompt` plus to the LLM cleanup prompt; bulk import via paste. +- **Live streaming transcription** with speech-gated chunking, hallucination filtering, and duplicate-boundary detection. + +### LLM formatting (local only) +- Local LLM runtime via [llama-cpp-2](https://github.com/utilityai/llama-cpp-rs) 0.1.144 with Vulkan. +- Three Qwen3 tiers (1.7B, 4B-Instruct-2507, 14B) auto-selected by hardware probe. +- GBNF grammar-constrained output for task extraction (always-parseable JSON). +- System prompt hardened against voice-delivered prompt injection. + +### Task capture +- Automatic task extraction from any transcript. +- **MicroSteps** — one-tap "break this task into 3–7 concrete physical actions." +- Profile-scoped task lists with inbox / today / soon / later buckets. +- Tasks back-link to their source transcript. + +### Input, paste, and window management +- **Global hotkey** — evdev-based on Linux (Wayland-compatible out of the box), `tauri-plugin-global-shortcut` on macOS / Windows. Per-OS capability matrix rejects invalid key combinations. +- **Platform-aware paste matrix** — `wtype` / `xdotool` / `ydotool` on Linux, AppleScript on macOS, SendKeys on Windows. Clipboard snapshot + 300 ms restore after paste. +- **Wayland-hardened transcription preview overlay** (`/preview`): pinned across virtual desktops, hidden from Alt+Tab via `WindowTypeHint::Utility`, never steals focus, focus-gated open. +- **Meeting auto-capture** (opt-in, default off): single-signal process-list watcher, user-editable app list, surfaces a non-modal reminder. No mic-activity heuristics, no calendar integration. + +### History and search +- **FTS5-indexed transcript search** over SQLite. +- **YAML-frontmatter markdown export** one-click into Obsidian vault. +- Per-transcript metadata: starred, manual tags, template, language, duration. +- Transcript editor window (`/viewer`) with debounced autosave. + +### External integration +- **MCP stdio server** (`kon-mcp`) exposing read-only transcripts and tasks to any Model Context Protocol client (Claude Desktop, Cline, Cursor, etc.). No authentication, read-only, local-only. + +### Accessibility +- Dyslexia-friendly fonts bundled: Lexend, Atkinson Hyperlegible Next, OpenDyslexic. +- Bionic reading mode. +- Per-region font size, letter spacing, line height, transcript-specific sizing. +- System-aware reduce-motion. +- **i18n**: English, Spanish, German (svelte-i18n scaffold). + +### Privacy, deployment, reliability +- Zero telemetry. Zero analytics. No crash reports leave the machine unless explicitly bundled. +- Auto-update via Tauri updater plugin (signed, user-approved). +- Per-window size + position persistence (`tauri-plugin-window-state`). +- Crash + panic capture stored locally; user-bundleable for support. + +--- + +## Architecture + +Kon is a Tauri 2 desktop app with three layers: + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Svelte 5 frontend (src/) │ +│ Routes: /, /float, /viewer, /preview │ +│ Stores, i18n, Tailwind CSS │ +├─────────────────────────────────────────────────────────────────┤ +│ Tauri 2 runtime (src-tauri/) │ +│ Commands: audio, clipboard, diagnostics, hotkey, live, llm, │ +│ meeting, models, paste, power, profiles, tasks, │ +│ transcription, transcripts, update, windows │ +│ Plugins: global-shortcut, dialog, opener, updater, │ +│ window-state │ +├─────────────────────────────────────────────────────────────────┤ +│ Rust workspace (crates/) │ +│ kon-core, kon-audio, kon-transcription, kon-llm, │ +│ kon-ai-formatting, kon-storage, kon-hotkey, │ +│ kon-cloud-providers, kon-mcp │ +└─────────────────────────────────────────────────────────────────┘ +``` + +The Rust workspace is the brain; Tauri is the OS integration surface; Svelte is the UI. The MCP server (`kon-mcp`) is a separate binary that opens Kon's SQLite store read-only — it's Kon-as-primitive for external agents. + +### Repository layout + +``` +kon/ +├── Cargo.toml # workspace root +├── src-tauri/ # Tauri app (main binary + commands) +│ ├── src/ +│ │ ├── commands/ # 18 Tauri command modules +│ │ ├── lib.rs # app entry, setup, command registration +│ │ ├── tray.rs +│ │ └── main.rs +│ ├── capabilities/ # Tauri ACL capability files +│ ├── gen/schemas/ # auto-generated ACL schemas +│ ├── tauri.conf.json # base Tauri config +│ ├── tauri.linux.conf.json # Linux overlay (native decorations) +│ └── resources/windows/ # Windows-specific bundled assets +├── crates/ # workspace Rust crates +│ ├── ai-formatting/ # post-processing pipeline + LLM cleanup client +│ ├── audio/ # capture, resampling, decoding, WAV I/O +│ ├── cloud-providers/ # BYOK cloud STT stubs (empty scaffolding) +│ ├── core/ # types, hardware probe, model registry, process watch +│ ├── hotkey/ # Linux evdev hotkey listener +│ ├── llm/ # llama-cpp-2 engine + model manager +│ ├── mcp/ # MCP stdio server binary +│ ├── storage/ # SQLite + FTS5 + file storage +│ └── transcription/ # Whisper + Parakeet wrappers, model mgmt +├── src/ # Svelte frontend +│ ├── routes/ # SvelteKit routes +│ │ ├── +page.svelte # main dictation UI +│ │ ├── +layout.svelte # shell (sidebar, tray sync, hotkey wiring) +│ │ ├── float/ # tasks float window +│ │ ├── viewer/ # transcript editor window +│ │ └── preview/ # transcription preview overlay +│ ├── lib/ +│ │ ├── pages/ # DictationPage, SettingsPage, HistoryPage, TasksPage, FilesPage, FirstRunPage +│ │ ├── components/ # reusable Svelte components +│ │ ├── stores/ # $state stores (page, preferences, profiles, toasts) +│ │ ├── actions/ # Svelte actions (bionic reading, etc.) +│ │ ├── utils/ # frontmatter, textMeasure, errors, storage helpers +│ │ ├── types/ # TS type definitions +│ │ └── i18n/ # svelte-i18n setup + en/es/de locales +│ ├── fonts/ # bundled accessibility fonts +│ ├── design-system/ # design tokens + UI kit references (not live code) +│ └── app.css +├── docs/ # all project documentation (see below) +├── .github/workflows/ # CI (check.yml, build.yml) +├── package.json +├── HANDOVER.md # latest session handover +└── run.sh # dev launcher (starts Vite then Tauri) +``` + +### Rust crates + +| Crate | Responsibility | +|---|---| +| **`kon-core`** | Shared types (`Segment`, `Transcript`, `Megabytes`, `ModelId`), constants, the `Engine` / `SpeedTier` / `AccuracyTier` enums, hardware probe (`sysinfo`-based), model registry (Whisper + Parakeet + Moonshine entries), hardware-aware recommendation scoring, `process_watch` for meeting detection. | +| **`kon-audio`** | `cpal`-based microphone capture with device hotplug + error forwarding, VAD, `rubato` streaming resampler to 16 kHz mono, `symphonia` file decoding, `hound` WAV I/O. | +| **`kon-transcription`** | `whisper-rs` backend (`WhisperRsBackend`) that owns a `WhisperContext` and supports `set_initial_prompt`. `LocalEngine` wraps both Whisper and Parakeet (via `transcribe-rs` ONNX) behind a common `SpeechToText` trait. Model manager handles downloads, paths, and disk checks. | +| **`kon-llm`** | `llama-cpp-2` engine with Qwen3 model manager. Three high-level surfaces: `cleanup_text` (formatting), `decompose_task` (3–7 micro-steps, GBNF-constrained JSON array), `extract_tasks` (optional-array, GBNF-constrained). Resumable HTTP downloads with SHA-256 verify. | +| **`kon-ai-formatting`** | Post-processing pipeline: filler removal, British English conversion, anti-hallucination filter, smart paragraph breaks on long pauses, optional LLM cleanup. Also hosts the `llm_client::CLEANUP_PROMPT` constant (prompt-injection-hardened). | +| **`kon-storage`** | SQLite via `sqlx` 0.8. Migrations, CRUD for transcripts / tasks / subtasks / profiles / profile terms / settings / error log, FTS5 search, file-storage paths. | +| **`kon-hotkey`** | Linux `evdev` hotkey listener with device hotplug. Parses Tauri-style hotkey strings (`Ctrl+Shift+R`), emits Pressed / Released events. Works natively on Wayland (no X11 dependency). Checks `/dev/input/event*` access on startup; surfaces a clear "add yourself to the `input` group" error when missing. | +| **`kon-cloud-providers`** | BYOK cloud-STT provider stubs. Currently empty scaffolding. When populated: OpenAI-compatible endpoint + Anthropic (ceiling for scope). | +| **`kon-mcp`** | Standalone `kon-mcp` binary implementing the MCP stdio protocol (2024-11-05). Read-only tools: `list_transcripts`, `get_transcript`, `search_transcripts`, `list_tasks`. Opens Kon's SQLite store. | + +### Tauri commands (src-tauri/src/commands/) + +| Module | What it exposes | +|---|---| +| `audio` | Device enumeration, native capture start/stop, audio-samples persistence | +| `clipboard` | Cross-platform clipboard write (arboard) | +| `diagnostics` | Panic hook, frontend error log, crash file listing, diagnostic report bundler | +| `hardware` | `probe_system`, `rank_models` | +| `hotkey` | `start_evdev_hotkey`, `update_evdev_hotkey`, `stop_evdev_hotkey`, `check_hotkey_access`, `is_wayland_session` | +| `live` | Live streaming transcription session lifecycle + speech-gate tuning | +| `llm` | Tier recommend, model check / download / load / unload / delete, status, `cleanup_transcript_text_cmd`, `extract_tasks_from_transcript_cmd` | +| `meeting` | `detect_meeting_processes` (process-list poll) | +| `models` | Whisper + Parakeet model download / load / check / default-id resolution, runtime capabilities API, pre-warm | +| `paste` | `paste_text` (copy + keystroke), `detect_paste_backends`, Wayland focus-race mitigation against the preview overlay | +| `power` | macOS `PowerAssertion` guard during long sessions (blocks App Nap) | +| `profiles` | Profile CRUD, profile-terms CRUD, learn-terms-from-edit | +| `tasks` | Task CRUD, subtask CRUD, `decompose_and_store`, `extract_tasks_from_transcript_cmd` | +| `transcription` | `transcribe_pcm`, `transcribe_file`, `transcribe_pcm_parakeet` | +| `transcripts` | Transcript CRUD + FTS5 search | +| `update` | Tauri-plugin-updater check / install | +| `windows` | `open_task_window`, `open_viewer_window`, `open_preview_window`, `close_preview_window` | + +### Frontend (src/) + +- **SvelteKit + Svelte 5 runes** (`$state`, `$derived`, `$effect`). +- **Tailwind CSS 4** for styling, with a Lexend/Atkinson/OpenDyslexic type system. +- **Secondary windows** (`/float`, `/viewer`, `/preview`) use named layouts (`+layout@.svelte`) to skip the main shell and run chrome-free. +- **Reactive stores** (`src/lib/stores/page.svelte.ts`): `settings`, `profiles`, `tasks`, `history`, `taskLists`, `templates`, `page`, `toasts`, `preferences`. +- **i18n**: `svelte-i18n` with en/es/de locales at `src/lib/i18n/locales/`. Scaffolding only — strings migrate to translation keys incrementally. + +--- + +## Runtime stack + +| Layer | Technology | Version | +|---|---|---| +| Desktop framework | [Tauri](https://tauri.app) | 2.10.3 | +| Frontend | Svelte 5 + SvelteKit + Vite | latest | +| Styling | Tailwind CSS | 4.x | +| Speech-to-text (primary) | whisper.cpp via [`whisper-rs`](https://github.com/tazz4843/whisper-rs) | 0.16 (Vulkan feature) | +| Speech-to-text (Parakeet) | sherpa-onnx via `transcribe-rs` | 0.3 | +| Local LLM | [`llama-cpp-2`](https://github.com/utilityai/llama-cpp-rs) | 0.1.144 (openmp + vulkan) | +| Database | SQLite via [`sqlx`](https://github.com/launchbadge/sqlx) | 0.8 | +| Async runtime | [`tokio`](https://tokio.rs/) | 1.x | +| Audio capture | [`cpal`](https://github.com/RustAudio/cpal) | current | +| Resampling | [`rubato`](https://github.com/HEnquist/rubato) | current | +| File decode | [`symphonia`](https://github.com/pdeljanov/Symphonia) | current | + +--- + +## Platform support + +| Platform | Status | Notes | +|---|---|---| +| Linux Wayland (KDE Plasma, GNOME Mutter, Hyprland, Sway) | **Primary target**, daily-dogfooded on KDE | evdev hotkey, GTK 3 via webkit2gtk, Vulkan, all paste backends | +| Linux X11 | Supported | xdotool paste path, GTK 3 | +| macOS | In CI, untested runtime | osascript paste, Metal via MoltenVK, App Nap guard | +| Windows | In CI, untested runtime | SendKeys paste, Vulkan-first GPU path, bundled DLLs for CPU fallback | + +CI runs `cargo check --workspace --all-targets` + `svelte-check` on all three on every push and PR. + +--- + +## Build + development + +### Prerequisites + +Linux (Fedora/RHEL listed; adjust for your distro): +``` +sudo dnf install libclang-devel clang \ + webkit2gtk4.1-devel libappindicator-gtk3-devel librsvg2-devel \ + alsa-lib-devel systemd-devel cmake \ + vulkan-headers vulkan-loader-devel glslc +``` + +macOS: +``` +brew install cmake llvm vulkan-headers vulkan-loader molten-vk shaderc +``` + +Windows: +``` +choco install cmake llvm vulkan-sdk +``` + +See [`docs/dev-setup.md`](docs/dev-setup.md) for the authoritative per-platform dependency list and for how `LIBCLANG_PATH` should be set. + +### Dev launch + +The fast path — starts Vite, waits for port 1420, then launches Tauri: + +```bash +./run.sh +``` + +Or manually: + +```bash +# Terminal 1 +npm run dev:frontend + +# Terminal 2 +npm run tauri dev +``` + +### Build + +```bash +npm run tauri build # release build, produces .AppImage / .deb / .dmg / .msi / .exe +``` + +CI also builds release installers on tag push (see `.github/workflows/build.yml`). + +### Testing + +```bash +cargo test --workspace --lib # 136 tests across 10 crates +npm run check # svelte-check (type-checks .svelte files) +cargo check --workspace --all-targets +``` + +--- + +## Project documentation + +Beyond this README, the repo ships extensive internal documentation: + +### Product + strategy — `docs/brief/` +Research briefs, competitive analysis, and strategic framing. Start with: +- [`what-kon-is.md`](docs/brief/what-kon-is.md) — product thesis +- [`why-current-tools-fail.md`](docs/brief/why-current-tools-fail.md) — market gap +- [`design-principles.md`](docs/brief/design-principles.md) — full principle list +- [`target-audience.md`](docs/brief/target-audience.md), [`market-size-demographics.md`](docs/brief/market-size-demographics.md) +- Appendices on cognitive ergonomics, AI body doubling, evolutionary psychology, implementation intentions, HITL scaffolding, voice interfaces + +### Brand — `docs/brand/` +- [`kon-brand-guidelines.md`](docs/brand/kon-brand-guidelines.md) +- [`kon-brand-platform.md`](docs/brand/kon-brand-platform.md) + +### Technical research — `docs/whisper-ecosystem/` +Cross-repo survey of 10 OSS Whisper projects, the Kon-specific atomic task backlog, and the two Cursor workstream plans. +- [`brief.md`](docs/whisper-ecosystem/brief.md) — 31-item task backlog (the canonical research spec) +- [`kon-context.md`](docs/whisper-ecosystem/kon-context.md) — ideology, shipped state, file-ownership fence for cloud AI agents +- [`workstream-A.md`](docs/whisper-ecosystem/workstream-A.md), [`workstream-B.md`](docs/whisper-ecosystem/workstream-B.md) — executed workstream plans + +### GPU tuning — `docs/gpu-tuning/` +- [`plan.md`](docs/gpu-tuning/plan.md) — MVP plan for GGML env-var panel + `kon-bench` auto-tuner + `kon-configs` community repo + +### Session handovers +- [`HANDOVER.md`](HANDOVER.md) — latest session summary +- Dated historical handovers: `HANDOVER-2026-04-17.md`, `HANDOVER-2026-04-18.md` + +### Dev reference +- [`docs/dev-setup.md`](docs/dev-setup.md) — dependency + launch reference +- [`docs/icon-mapping.md`](docs/icon-mapping.md) — icon conventions + +--- + +## Roadmap + +The shipped code represents Phases 1–3 and a partial Phase 4. + +Pinned roadmap items (scoped in docs and session memory): + +- **Phase 4** — remaining items from [`workstream-A.md`](docs/whisper-ecosystem/workstream-A.md) + [`workstream-B.md`](docs/whisper-ecosystem/workstream-B.md) +- **Voice calibration** — three-tier plan replacing the hardcoded speech-gate with per-user baselines +- **GPU community tuning** — see [`docs/gpu-tuning/plan.md`](docs/gpu-tuning/plan.md); five-phase roadmap from settings panel to agentic auto-tuner + community config repo +- **Cloud endpoint contract test** — when `kon-cloud-providers` grows a real provider +- **`ggml` dedup** — replace the interim `-Wl,--allow-multiple-definition` link flag with a proper shared-lib setup; unblocks custom shader / backend work +- **Mobile (iOS / Android)** — long-horizon, gated on the single-binary Rust stack scaling + +Explicitly shelved (not coming without specific community signal): +- Wake-word / always-listening agent +- Chat-style LLM UI +- Multi-provider cloud fan-out beyond OpenAI-compatible + Anthropic +- Second notes-editing surface (transcripts leave Kon via frontmatter to Obsidian) +- Speaker diarization +- Dragon-style passage-based speaker fine-tuning (Whisper has no speaker adaptation) + +--- + +## Contributing + +Pre-alpha status; contribution process TBD before public beta. For now: + +- Every Tauri command change must register in both [`src-tauri/src/lib.rs`](src-tauri/src/lib.rs) (invoke handler) and in the invoking frontend code. +- Every Settings-visible setting must have a type field in [`src/lib/types/app.ts`](src/lib/types/app.ts) and a default in [`src/lib/stores/page.svelte.ts`](src/lib/stores/page.svelte.ts). +- Every new workspace crate needs a `description` in its `Cargo.toml`. +- Tests: add at least a smoke test per new Tauri command or crate module. The workspace test floor is "no regressions on main." +- Wayland compatibility is a first-class concern — don't assume X11. The preview overlay and paste matrix live-document what this looks like in practice. + +--- + +## Licence + +To be finalised before public beta. Current intent: MIT or similar permissive licence, with Corbel Consulting offering optional commercial support / managed services as the revenue path. + +--- + +## Contact + +**Jake Sames** — [jakeadriansames@gmail.com](mailto:jakeadriansames@gmail.com) +Repo: [github.com/jakejars/kon](https://github.com/jakejars/kon) · [git.corbel.consulting/jake/kon](https://git.corbel.consulting/jake/kon)