feat(plan): add Phase 2 functional MVP implementation plan
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
902
docs/superpowers/plans/2026-03-21-phase2-functional-mvp.md
Normal file
902
docs/superpowers/plans/2026-03-21-phase2-functional-mvp.md
Normal file
@@ -0,0 +1,902 @@
|
|||||||
|
# Kon Phase 2: Functional MVP — Implementation Plan
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||||
|
|
||||||
|
**Goal:** Transform Kon from a branded shell into a functional voice → text → tasks pipeline with local LLM intelligence, delivering a shippable closed-beta desktop app.
|
||||||
|
|
||||||
|
**Architecture:** The existing codebase has a working audio capture → Whisper transcription → text display pipeline via browser AudioWorklet + Tauri IPC. Phase 2 migrates persistence from localStorage to SQLite (backend already has schema + CRUD), adds FTS5 search, wires llama-cpp-2 for local LLM task extraction and micro-stepping, connects the VisualTimer to tasks, and polishes first-run + settings + export.
|
||||||
|
|
||||||
|
**Tech Stack:** Svelte 5, SvelteKit 2, Tailwind CSS 4.2, Tauri 2, Rust, sqlx (SQLite), whisper-rs (via transcribe-rs), llama-cpp-2, lucide-svelte
|
||||||
|
|
||||||
|
**Branch:** `phase-2/functional-mvp`
|
||||||
|
|
||||||
|
**Commit format:** `feat(scope): description`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Existing State Summary
|
||||||
|
|
||||||
|
### Already Working
|
||||||
|
- Microphone capture via browser AudioWorklet → 16kHz mono PCM
|
||||||
|
- Whisper + Parakeet transcription via transcribe-rs (streaming chunks)
|
||||||
|
- Model download/load/cache management
|
||||||
|
- Text post-processing (filler removal, British English, anti-hallucination)
|
||||||
|
- Rule-based task extraction (frontend JS — `taskExtractor.js`)
|
||||||
|
- Task CRUD in localStorage with BroadcastChannel multi-window sync
|
||||||
|
- History in localStorage with playback
|
||||||
|
- File transcription (drag-drop, multi-format)
|
||||||
|
- Preferences store with SQLite persistence
|
||||||
|
- Full brand token system, accessibility controls, sensory zones
|
||||||
|
|
||||||
|
### Needs Building
|
||||||
|
1. **SQLite migration v2**: Add `priority`, `project`, `status`, `updated_at` to tasks; add FTS5 virtual table for transcripts
|
||||||
|
2. **Tauri commands for task CRUD**: Replace localStorage task management with SQLite backend
|
||||||
|
3. **Tauri commands for transcript persistence**: Save transcriptions to SQLite (currently only localStorage)
|
||||||
|
4. **FTS5 full-text search**: Backend search across transcriptions
|
||||||
|
5. **llama-cpp-2 integration**: Wire LLM inference engine for task extraction + micro-stepping
|
||||||
|
6. **LLM model management**: Download/cache GGUF models (Phi-4-mini, Qwen 3 7B)
|
||||||
|
7. **Micro-stepping UI**: Inline micro-steps below parent tasks with "Just Start" timer
|
||||||
|
8. **VisualTimer wiring**: Connect timer to tasks, add notifications
|
||||||
|
9. **Export to Obsidian**: Markdown with YAML frontmatter
|
||||||
|
10. **Global hotkey update**: Change default from Ctrl+Shift+R to Ctrl+Shift+Space
|
||||||
|
11. **Settings backend wiring**: Migrate remaining settings to SQLite preferences
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## File Map
|
||||||
|
|
||||||
|
### New files to create
|
||||||
|
|
||||||
|
| File | Purpose |
|
||||||
|
|---|---|
|
||||||
|
| `crates/ai-formatting/src/llm_client.rs` | llama-cpp-2 inference wrapper (rewrite from placeholder) |
|
||||||
|
| `crates/ai-formatting/src/task_extraction.rs` | LLM-based task extraction with fallback to rule-based |
|
||||||
|
| `crates/ai-formatting/src/micro_stepping.rs` | Task decomposition into micro-steps |
|
||||||
|
| `crates/llm/Cargo.toml` | New crate for LLM model management |
|
||||||
|
| `crates/llm/src/lib.rs` | LLM engine wrapper |
|
||||||
|
| `crates/llm/src/model_manager.rs` | GGUF model download/cache |
|
||||||
|
| `crates/llm/src/inference.rs` | Token streaming inference |
|
||||||
|
| `src-tauri/src/commands/tasks.rs` | Task CRUD Tauri commands |
|
||||||
|
| `src-tauri/src/commands/history.rs` | Transcript persistence + FTS5 search commands |
|
||||||
|
| `src-tauri/src/commands/llm.rs` | LLM model management + inference commands |
|
||||||
|
| `src/lib/components/MicroSteps.svelte` | Micro-step display + "Just Start" button |
|
||||||
|
| `src/lib/components/TaskTimer.svelte` | Timer wired to specific task |
|
||||||
|
| `src/lib/stores/tasks.svelte.js` | Task store backed by SQLite via Tauri commands |
|
||||||
|
| `src/lib/stores/history.svelte.js` | History store backed by SQLite |
|
||||||
|
| `src/lib/utils/obsidianExport.js` | Obsidian vault export logic |
|
||||||
|
|
||||||
|
### Files to modify
|
||||||
|
|
||||||
|
| File | Changes |
|
||||||
|
|---|---|
|
||||||
|
| `crates/storage/src/migrations.rs` | Add migration v2 (FTS5, task columns, timer state) |
|
||||||
|
| `crates/storage/src/database.rs` | Add task CRUD with new columns, FTS5 search, timer persistence |
|
||||||
|
| `crates/ai-formatting/Cargo.toml` | Add serde, serde_json dependencies |
|
||||||
|
| `src-tauri/Cargo.toml` | Add llama-cpp-2, tauri-plugin-notification |
|
||||||
|
| `src-tauri/src/lib.rs` | Register new commands, add LLM state |
|
||||||
|
| `src-tauri/src/commands/mod.rs` | Add new command modules |
|
||||||
|
| `src/lib/pages/DictationPage.svelte` | Wire SQLite transcript persistence |
|
||||||
|
| `src/lib/pages/TasksPage.svelte` | Wire SQLite task CRUD, add micro-steps |
|
||||||
|
| `src/lib/pages/HistoryPage.svelte` | Wire FTS5 search, SQLite history |
|
||||||
|
| `src/lib/pages/FilesPage.svelte` | Wire SQLite persistence for file transcriptions |
|
||||||
|
| `src/lib/pages/FirstRunPage.svelte` | Add LLM model download step |
|
||||||
|
| `src/lib/pages/SettingsPage.svelte` | Wire remaining settings to backend |
|
||||||
|
| `src/lib/stores/page.svelte.js` | Remove localStorage task/history stores (migrate to new stores) |
|
||||||
|
| `src/lib/components/WipTaskList.svelte` | Add micro-step expansion, timer button |
|
||||||
|
| `src/lib/components/VisualTimer.svelte` | Add countdown logic, notifications |
|
||||||
|
| `src/lib/components/ModelDownloader.svelte` | Support LLM model downloads |
|
||||||
|
| `Cargo.toml` | Add crates/llm to workspace |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 2A — Core Pipeline
|
||||||
|
|
||||||
|
### Task 1: SQLite Migration v2 — Schema Extensions
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `crates/storage/src/migrations.rs`
|
||||||
|
- Modify: `crates/storage/src/database.rs`
|
||||||
|
- Modify: `crates/storage/Cargo.toml`
|
||||||
|
|
||||||
|
**Why first:** Everything else depends on the database schema being right.
|
||||||
|
|
||||||
|
- [ ] **Step 1: Add migration v2 to migrations.rs**
|
||||||
|
|
||||||
|
Add after the existing migration v1 entry in the `MIGRATIONS` array:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
(2, "phase 2 — task fields, FTS5, timer state", r#"
|
||||||
|
ALTER TABLE tasks ADD COLUMN priority TEXT NOT NULL DEFAULT 'medium';
|
||||||
|
ALTER TABLE tasks ADD COLUMN project TEXT;
|
||||||
|
ALTER TABLE tasks ADD COLUMN status TEXT NOT NULL DEFAULT 'pending';
|
||||||
|
ALTER TABLE tasks ADD COLUMN updated_at TEXT NOT NULL DEFAULT (datetime('now'));
|
||||||
|
ALTER TABLE tasks ADD COLUMN sort_order INTEGER NOT NULL DEFAULT 0;
|
||||||
|
ALTER TABLE tasks ADD COLUMN notes TEXT NOT NULL DEFAULT '';
|
||||||
|
|
||||||
|
CREATE VIRTUAL TABLE IF NOT EXISTS transcripts_fts USING fts5(
|
||||||
|
text,
|
||||||
|
title,
|
||||||
|
content='transcripts',
|
||||||
|
content_rowid='rowid'
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TRIGGER IF NOT EXISTS transcripts_ai AFTER INSERT ON transcripts BEGIN
|
||||||
|
INSERT INTO transcripts_fts(rowid, text, title)
|
||||||
|
VALUES (new.rowid, new.text, new.title);
|
||||||
|
END;
|
||||||
|
|
||||||
|
CREATE TRIGGER IF NOT EXISTS transcripts_ad AFTER DELETE ON transcripts BEGIN
|
||||||
|
INSERT INTO transcripts_fts(transcripts_fts, rowid, text, title)
|
||||||
|
VALUES ('delete', old.rowid, old.text, old.title);
|
||||||
|
END;
|
||||||
|
|
||||||
|
CREATE TRIGGER IF NOT EXISTS transcripts_au AFTER UPDATE ON transcripts BEGIN
|
||||||
|
INSERT INTO transcripts_fts(transcripts_fts, rowid, text, title)
|
||||||
|
VALUES ('delete', old.rowid, old.text, old.title);
|
||||||
|
INSERT INTO transcripts_fts(rowid, text, title)
|
||||||
|
VALUES (new.rowid, new.text, new.title);
|
||||||
|
END;
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS timer_state (
|
||||||
|
id TEXT PRIMARY KEY DEFAULT 'active',
|
||||||
|
task_id TEXT NOT NULL,
|
||||||
|
total_seconds INTEGER NOT NULL,
|
||||||
|
remaining_seconds INTEGER NOT NULL,
|
||||||
|
started_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||||
|
paused INTEGER NOT NULL DEFAULT 0
|
||||||
|
)
|
||||||
|
"#),
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Add new database functions to database.rs**
|
||||||
|
|
||||||
|
Add task functions with new columns:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
// Task CRUD with extended fields
|
||||||
|
pub async fn insert_task_v2(pool, id, text, priority, project, status, bucket, effort, source_transcript_id, sort_order) -> Result<()>
|
||||||
|
pub async fn update_task_v2(pool, id, text, priority, project, status, bucket, effort, notes) -> Result<()>
|
||||||
|
pub async fn reorder_tasks(pool, task_ids: &[String]) -> Result<()>
|
||||||
|
pub async fn list_tasks_by_status(pool, status, limit) -> Result<Vec<TaskRow>>
|
||||||
|
pub async fn search_transcripts(pool, query: &str, limit: i64) -> Result<Vec<TranscriptRow>>
|
||||||
|
|
||||||
|
// Timer state persistence
|
||||||
|
pub async fn save_timer_state(pool, task_id, total_seconds, remaining_seconds, paused) -> Result<()>
|
||||||
|
pub async fn get_timer_state(pool) -> Result<Option<TimerStateRow>>
|
||||||
|
pub async fn clear_timer_state(pool) -> Result<()>
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 3: Add FTS5 search function**
|
||||||
|
|
||||||
|
```rust
|
||||||
|
pub async fn search_transcripts(pool: &SqlitePool, query: &str, limit: i64) -> Result<Vec<TranscriptRow>> {
|
||||||
|
let rows = sqlx::query(
|
||||||
|
"SELECT t.id, t.text, t.source, t.title, t.audio_path, t.duration, t.engine, t.model_id, t.inference_ms, t.sample_rate, t.audio_channels, t.format_mode, t.remove_fillers, t.british_english, t.anti_hallucination, t.created_at
|
||||||
|
FROM transcripts t
|
||||||
|
JOIN transcripts_fts fts ON t.rowid = fts.rowid
|
||||||
|
WHERE transcripts_fts MATCH ?1
|
||||||
|
ORDER BY rank
|
||||||
|
LIMIT ?2"
|
||||||
|
)
|
||||||
|
.bind(query)
|
||||||
|
.bind(limit)
|
||||||
|
.fetch_all(pool)
|
||||||
|
.await
|
||||||
|
.map_err(|e| KonError::StorageError(format!("FTS search failed: {e}")))?;
|
||||||
|
Ok(rows.iter().map(transcript_row_from).collect())
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: Run tests**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd crates/storage && cargo test
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 5: Verify Tauri app compiles**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd src-tauri && cargo check
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 6: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add crates/storage/
|
||||||
|
git commit -m "feat(storage): add migration v2 — task fields, FTS5 search, timer state"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 2: Tauri Commands for Transcript Persistence
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `src-tauri/src/commands/history.rs`
|
||||||
|
- Modify: `src-tauri/src/commands/mod.rs`
|
||||||
|
- Modify: `src-tauri/src/lib.rs`
|
||||||
|
- Modify: `src-tauri/src/commands/transcription.rs`
|
||||||
|
|
||||||
|
- [ ] **Step 1: Create history.rs with transcript CRUD commands**
|
||||||
|
|
||||||
|
```rust
|
||||||
|
// save_transcript — persist completed transcription to SQLite
|
||||||
|
// get_transcript — fetch by ID
|
||||||
|
// list_transcripts — paginated list, newest first
|
||||||
|
// delete_transcript — remove by ID
|
||||||
|
// search_transcripts — FTS5 search
|
||||||
|
// save_segments — batch insert segments for a transcript
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Register commands in mod.rs and lib.rs**
|
||||||
|
|
||||||
|
- [ ] **Step 3: Modify transcription.rs to auto-persist**
|
||||||
|
|
||||||
|
After successful transcription, auto-save the transcript + segments to SQLite (in addition to emitting the event).
|
||||||
|
|
||||||
|
- [ ] **Step 4: Verify compilation**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd src-tauri && cargo check
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 5: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add src-tauri/
|
||||||
|
git commit -m "feat(history): add Tauri commands for transcript persistence and FTS5 search"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 3: Tauri Commands for Task CRUD
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `src-tauri/src/commands/tasks.rs`
|
||||||
|
- Modify: `src-tauri/src/commands/mod.rs`
|
||||||
|
- Modify: `src-tauri/src/lib.rs`
|
||||||
|
|
||||||
|
- [ ] **Step 1: Create tasks.rs**
|
||||||
|
|
||||||
|
Commands:
|
||||||
|
```rust
|
||||||
|
#[tauri::command] async fn create_task(state, text, priority, project, bucket, effort, source_transcript_id) -> Result<TaskResponse, String>
|
||||||
|
#[tauri::command] async fn update_task(state, id, text, priority, project, status, bucket, effort, notes) -> Result<(), String>
|
||||||
|
#[tauri::command] async fn delete_task(state, id) -> Result<(), String>
|
||||||
|
#[tauri::command] async fn list_tasks(state, status, limit) -> Result<Vec<TaskResponse>, String>
|
||||||
|
#[tauri::command] async fn reorder_tasks(state, task_ids: Vec<String>) -> Result<(), String>
|
||||||
|
#[tauri::command] async fn complete_task(state, id) -> Result<(), String>
|
||||||
|
```
|
||||||
|
|
||||||
|
TaskResponse struct:
|
||||||
|
```rust
|
||||||
|
#[derive(Serialize)]
|
||||||
|
struct TaskResponse {
|
||||||
|
id: String,
|
||||||
|
text: String,
|
||||||
|
priority: String,
|
||||||
|
project: Option<String>,
|
||||||
|
status: String,
|
||||||
|
bucket: String,
|
||||||
|
effort: Option<String>,
|
||||||
|
done: bool,
|
||||||
|
done_at: Option<String>,
|
||||||
|
created_at: String,
|
||||||
|
updated_at: String,
|
||||||
|
sort_order: i64,
|
||||||
|
notes: String,
|
||||||
|
source_transcript_id: Option<String>,
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Register commands in mod.rs and lib.rs**
|
||||||
|
|
||||||
|
- [ ] **Step 3: Verify compilation**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd src-tauri && cargo check
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add src-tauri/
|
||||||
|
git commit -m "feat(tasks): add Tauri commands for full task CRUD with priority, project, status"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 4: Frontend Task Store Migration (localStorage → SQLite)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `src/lib/stores/tasks.svelte.js`
|
||||||
|
- Modify: `src/lib/pages/TasksPage.svelte`
|
||||||
|
- Modify: `src/lib/components/WipTaskList.svelte`
|
||||||
|
- Modify: `src/lib/stores/page.svelte.js`
|
||||||
|
|
||||||
|
- [ ] **Step 1: Create tasks.svelte.js**
|
||||||
|
|
||||||
|
New store that wraps Tauri commands instead of localStorage:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
import { invoke } from '@tauri-apps/api/core';
|
||||||
|
|
||||||
|
let tasks = $state([]);
|
||||||
|
let loading = $state(false);
|
||||||
|
|
||||||
|
export async function loadTasks() { ... }
|
||||||
|
export async function createTask(text, opts = {}) { ... }
|
||||||
|
export async function updateTask(id, updates) { ... }
|
||||||
|
export async function deleteTask(id) { ... }
|
||||||
|
export async function completeTask(id) { ... }
|
||||||
|
export async function reorderTasks(ids) { ... }
|
||||||
|
export function getTasks() { return tasks; }
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Update TasksPage.svelte to use new store**
|
||||||
|
|
||||||
|
Replace all `tasks` imports from page.svelte.js with the new SQLite-backed store.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Update WipTaskList.svelte**
|
||||||
|
|
||||||
|
Wire to new task store.
|
||||||
|
|
||||||
|
- [ ] **Step 4: Keep page.svelte.js tasks for backwards compat during migration**
|
||||||
|
|
||||||
|
Add a bridge that loads from SQLite on mount, falls back to localStorage.
|
||||||
|
|
||||||
|
- [ ] **Step 5: Verify build**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm run build
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 6: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add src/
|
||||||
|
git commit -m "feat(tasks): migrate task store from localStorage to SQLite backend"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 5: Frontend History Store Migration
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `src/lib/stores/history.svelte.js`
|
||||||
|
- Modify: `src/lib/pages/HistoryPage.svelte`
|
||||||
|
- Modify: `src/lib/pages/DictationPage.svelte`
|
||||||
|
|
||||||
|
- [ ] **Step 1: Create history.svelte.js**
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
import { invoke } from '@tauri-apps/api/core';
|
||||||
|
|
||||||
|
let transcripts = $state([]);
|
||||||
|
|
||||||
|
export async function loadHistory(limit = 100) { ... }
|
||||||
|
export async function saveTranscript(transcript) { ... }
|
||||||
|
export async function deleteTranscript(id) { ... }
|
||||||
|
export async function searchTranscripts(query) { ... }
|
||||||
|
export function getHistory() { return transcripts; }
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Update HistoryPage.svelte**
|
||||||
|
|
||||||
|
Replace localStorage-based history with SQLite search. Wire FTS5 search to the search input.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Update DictationPage.svelte**
|
||||||
|
|
||||||
|
After transcription completes, call `saveTranscript()` from the new store (in addition to existing behaviour).
|
||||||
|
|
||||||
|
- [ ] **Step 4: Verify build**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm run build
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 5: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add src/
|
||||||
|
git commit -m "feat(history): migrate history to SQLite with FTS5 search"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 2B — Intelligence Layer
|
||||||
|
|
||||||
|
### Task 6: LLM Crate + llama-cpp-2 Integration
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `crates/llm/Cargo.toml`
|
||||||
|
- Create: `crates/llm/src/lib.rs`
|
||||||
|
- Create: `crates/llm/src/inference.rs`
|
||||||
|
- Create: `crates/llm/src/model_manager.rs`
|
||||||
|
- Modify: `Cargo.toml` (workspace members)
|
||||||
|
- Modify: `src-tauri/Cargo.toml` (add dependency)
|
||||||
|
|
||||||
|
**Note:** llama-cpp-2 requires CMake and a C++ compiler. On Windows this means MSVC build tools.
|
||||||
|
|
||||||
|
- [ ] **Step 1: Create crates/llm/Cargo.toml**
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[package]
|
||||||
|
name = "kon-llm"
|
||||||
|
version = "0.1.0"
|
||||||
|
edition = "2021"
|
||||||
|
description = "Local LLM inference via llama.cpp for Kon"
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
kon-core = { path = "../core" }
|
||||||
|
llama-cpp-2 = { version = "0.1", features = ["vulkan"] }
|
||||||
|
tokio = { version = "1", features = ["rt", "sync"] }
|
||||||
|
reqwest = { version = "0.12", features = ["stream"] }
|
||||||
|
futures-util = "0.3"
|
||||||
|
serde = { version = "1", features = ["derive"] }
|
||||||
|
serde_json = "1"
|
||||||
|
log = "0.4"
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Create lib.rs with LlmEngine struct**
|
||||||
|
|
||||||
|
```rust
|
||||||
|
pub struct LlmEngine {
|
||||||
|
model: Mutex<Option<LlamaModel>>,
|
||||||
|
loaded_model_path: Mutex<Option<PathBuf>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl LlmEngine {
|
||||||
|
pub fn new() -> Self { ... }
|
||||||
|
pub fn load(&self, model_path: &Path) -> Result<()> { ... }
|
||||||
|
pub fn is_loaded(&self) -> bool { ... }
|
||||||
|
pub fn generate(&self, prompt: &str, max_tokens: u32) -> Result<String> { ... }
|
||||||
|
pub fn generate_streaming(&self, prompt: &str, max_tokens: u32, callback: impl Fn(&str)) -> Result<String> { ... }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 3: Create model_manager.rs for GGUF downloads**
|
||||||
|
|
||||||
|
Reuse the pattern from crates/transcription/model_manager.rs — streaming download with progress callback, atomic rename.
|
||||||
|
|
||||||
|
Model catalog:
|
||||||
|
```rust
|
||||||
|
const LLM_MODELS: &[LlmModelEntry] = &[
|
||||||
|
LlmModelEntry {
|
||||||
|
id: "phi-4-mini-q4",
|
||||||
|
display_name: "Phi-4 Mini (8GB RAM)",
|
||||||
|
url: "https://huggingface.co/...",
|
||||||
|
disk_size: Megabytes(2300),
|
||||||
|
ram_required: Megabytes(4000),
|
||||||
|
filename: "phi-4-mini-q4_k_m.gguf",
|
||||||
|
},
|
||||||
|
LlmModelEntry {
|
||||||
|
id: "qwen3-7b-q4",
|
||||||
|
display_name: "Qwen 3 7B (16GB RAM)",
|
||||||
|
url: "https://huggingface.co/...",
|
||||||
|
disk_size: Megabytes(4500),
|
||||||
|
ram_required: Megabytes(8000),
|
||||||
|
filename: "qwen3-7b-q4_k_m.gguf",
|
||||||
|
},
|
||||||
|
];
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: Create inference.rs with async wrapper**
|
||||||
|
|
||||||
|
```rust
|
||||||
|
pub async fn run_llm_inference(
|
||||||
|
engine: Arc<LlmEngine>,
|
||||||
|
prompt: String,
|
||||||
|
max_tokens: u32,
|
||||||
|
) -> Result<String> {
|
||||||
|
tokio::task::spawn_blocking(move || {
|
||||||
|
engine.generate(&prompt, max_tokens)
|
||||||
|
}).await.map_err(|e| KonError::Other(e.to_string()))?
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 5: Add workspace member and verify compilation**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo check -p kon-llm
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 6: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add crates/llm/ Cargo.toml
|
||||||
|
git commit -m "feat(llm): add kon-llm crate with llama-cpp-2 inference engine"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 7: LLM Tauri Commands + Model Download UI
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `src-tauri/src/commands/llm.rs`
|
||||||
|
- Modify: `src-tauri/src/commands/mod.rs`
|
||||||
|
- Modify: `src-tauri/src/lib.rs`
|
||||||
|
- Modify: `src/lib/pages/SettingsPage.svelte`
|
||||||
|
- Modify: `src/lib/components/ModelDownloader.svelte`
|
||||||
|
- Modify: `src/lib/pages/FirstRunPage.svelte`
|
||||||
|
|
||||||
|
- [ ] **Step 1: Create llm.rs with commands**
|
||||||
|
|
||||||
|
```rust
|
||||||
|
#[tauri::command] async fn list_llm_models() -> Vec<LlmModelInfo>
|
||||||
|
#[tauri::command] async fn download_llm_model(app, id) -> Result<(), String> // emits "llm-download-progress"
|
||||||
|
#[tauri::command] async fn load_llm_model(state, id) -> Result<(), String>
|
||||||
|
#[tauri::command] async fn check_llm_engine(state) -> bool
|
||||||
|
#[tauri::command] async fn llm_generate(state, prompt, max_tokens) -> Result<String, String>
|
||||||
|
#[tauri::command] async fn extract_tasks_llm(state, transcript_text) -> Result<Vec<TaskSuggestion>, String>
|
||||||
|
#[tauri::command] async fn decompose_task(state, task_text) -> Result<Vec<MicroStep>, String>
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Add LlmEngine to AppState**
|
||||||
|
|
||||||
|
```rust
|
||||||
|
pub struct AppState {
|
||||||
|
pub whisper_engine: Arc<LocalEngine>,
|
||||||
|
pub parakeet_engine: Arc<LocalEngine>,
|
||||||
|
pub llm_engine: Arc<LlmEngine>,
|
||||||
|
pub db: SqlitePool,
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 3: Register commands in lib.rs**
|
||||||
|
|
||||||
|
- [ ] **Step 4: Update ModelDownloader.svelte to support LLM models**
|
||||||
|
|
||||||
|
Add a `modelType` prop ("whisper" | "llm") and listen to appropriate download events.
|
||||||
|
|
||||||
|
- [ ] **Step 5: Add LLM model section to FirstRunPage.svelte**
|
||||||
|
|
||||||
|
After STT model download, offer optional LLM model download: "Download AI assistant for task extraction? (optional, {size})"
|
||||||
|
|
||||||
|
- [ ] **Step 6: Add LLM section to SettingsPage.svelte**
|
||||||
|
|
||||||
|
In the "AI Assistant" accordion: model selection, download button, status indicator.
|
||||||
|
|
||||||
|
- [ ] **Step 7: Verify build**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd src-tauri && cargo check && cd .. && npm run build
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 8: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add src-tauri/ src/ crates/llm/
|
||||||
|
git commit -m "feat(llm): add LLM Tauri commands, model download UI, FirstRun integration"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 8: Task Extraction — LLM + Rule-Based Fallback
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Rewrite: `crates/ai-formatting/src/llm_client.rs` (replace placeholder)
|
||||||
|
- Create: `crates/ai-formatting/src/task_extraction.rs`
|
||||||
|
- Modify: `crates/ai-formatting/Cargo.toml`
|
||||||
|
- Modify: `crates/ai-formatting/src/lib.rs`
|
||||||
|
- Modify: `src-tauri/src/commands/llm.rs`
|
||||||
|
- Modify: `src/lib/pages/DictationPage.svelte`
|
||||||
|
|
||||||
|
- [ ] **Step 1: Create task_extraction.rs**
|
||||||
|
|
||||||
|
```rust
|
||||||
|
pub struct ExtractedTask {
|
||||||
|
pub title: String,
|
||||||
|
pub priority: String,
|
||||||
|
pub project: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
const EXTRACTION_SYSTEM_PROMPT: &str = r#"Extract actionable tasks from the following voice transcription. Each task must start with a concrete verb. Return as JSON array of {"title": "...", "priority": "high|medium|low", "project": "..."}.
|
||||||
|
Only extract genuine tasks — not observations or comments. If no tasks found, return empty array []."#;
|
||||||
|
|
||||||
|
pub fn extract_tasks_with_llm(engine: &LlmEngine, transcript: &str) -> Result<Vec<ExtractedTask>> { ... }
|
||||||
|
pub fn extract_tasks_rule_based(transcript: &str) -> Vec<ExtractedTask> { ... }
|
||||||
|
pub fn extract_tasks(engine: Option<&LlmEngine>, transcript: &str) -> Vec<ExtractedTask> { ... }
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Wire into extract_tasks_llm command**
|
||||||
|
|
||||||
|
The Tauri command tries LLM first, falls back to rule-based.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Update DictationPage.svelte**
|
||||||
|
|
||||||
|
Replace the JS `extractTasks()` call with `invoke('extract_tasks_llm', { transcriptText })`.
|
||||||
|
|
||||||
|
- [ ] **Step 4: Verify build**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd src-tauri && cargo check && cd .. && npm run build
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 5: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add crates/ai-formatting/ src-tauri/ src/
|
||||||
|
git commit -m "feat(extraction): add LLM task extraction with rule-based fallback"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 9: Micro-Stepping
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `crates/ai-formatting/src/micro_stepping.rs`
|
||||||
|
- Create: `src/lib/components/MicroSteps.svelte`
|
||||||
|
- Modify: `src/lib/components/WipTaskList.svelte`
|
||||||
|
- Modify: `src-tauri/src/commands/llm.rs`
|
||||||
|
|
||||||
|
- [ ] **Step 1: Create micro_stepping.rs**
|
||||||
|
|
||||||
|
```rust
|
||||||
|
const MICRO_STEP_PROMPT: &str = r#"Break this task into 3-7 micro-steps. Each step MUST start with a specific physical verb (e.g. 'Open', 'Type', 'Click', 'Pick up'). Each step must be completable in under 5 minutes. Never use abstract verbs like 'organise', 'plan', 'consider'. Return as JSON array of strings."#;
|
||||||
|
|
||||||
|
pub fn decompose_task(engine: &LlmEngine, task_text: &str) -> Result<Vec<String>> { ... }
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Wire into decompose_task Tauri command**
|
||||||
|
|
||||||
|
- [ ] **Step 3: Create MicroSteps.svelte**
|
||||||
|
|
||||||
|
```svelte
|
||||||
|
<script>
|
||||||
|
import { invoke } from '@tauri-apps/api/core';
|
||||||
|
import { Play } from 'lucide-svelte';
|
||||||
|
let { taskId, taskText } = $props();
|
||||||
|
let steps = $state([]);
|
||||||
|
let loading = $state(false);
|
||||||
|
// ...
|
||||||
|
</script>
|
||||||
|
```
|
||||||
|
|
||||||
|
Shows expandable micro-steps below a task. Each step has a "Just Start" button that launches a 2min or 5min timer.
|
||||||
|
|
||||||
|
- [ ] **Step 4: Wire MicroSteps into WipTaskList**
|
||||||
|
|
||||||
|
Add expand/collapse per task that loads micro-steps on demand.
|
||||||
|
|
||||||
|
- [ ] **Step 5: Verify build**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm run build
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 6: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add crates/ai-formatting/ src-tauri/ src/
|
||||||
|
git commit -m "feat(microsteps): add LLM task decomposition with Just Start timer"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 10: Visual Timer Wiring + Notifications
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `src/lib/components/VisualTimer.svelte`
|
||||||
|
- Create: `src/lib/components/TaskTimer.svelte`
|
||||||
|
- Modify: `src-tauri/Cargo.toml` (add tauri-plugin-notification)
|
||||||
|
- Modify: `src-tauri/src/lib.rs` (register notification plugin)
|
||||||
|
- Modify: `src-tauri/tauri.conf.json` (add notification permission)
|
||||||
|
|
||||||
|
- [ ] **Step 1: Add tauri-plugin-notification**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd src-tauri && cargo add tauri-plugin-notification@2
|
||||||
|
```
|
||||||
|
|
||||||
|
Update lib.rs: `.plugin(tauri_plugin_notification::init())`
|
||||||
|
|
||||||
|
Update tauri.conf.json capabilities.
|
||||||
|
|
||||||
|
- [ ] **Step 2: Create TaskTimer.svelte**
|
||||||
|
|
||||||
|
Wraps VisualTimer with countdown logic, persists timer state to SQLite, shows OS notification on complete:
|
||||||
|
|
||||||
|
```svelte
|
||||||
|
<script>
|
||||||
|
import VisualTimer from './VisualTimer.svelte';
|
||||||
|
import { invoke } from '@tauri-apps/api/core';
|
||||||
|
import { sendNotification } from '@tauri-apps/plugin-notification';
|
||||||
|
// Timer countdown, pause/resume, persist state
|
||||||
|
</script>
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 3: Wire timer persistence**
|
||||||
|
|
||||||
|
On start: `invoke('save_timer_state', { taskId, totalSeconds, remainingSeconds })`
|
||||||
|
On tick: Update remaining (debounced, every 5s)
|
||||||
|
On complete: `invoke('clear_timer_state')` + notification
|
||||||
|
On app restart: `invoke('get_timer_state')` → resume timer
|
||||||
|
|
||||||
|
- [ ] **Step 4: Respect reduce-motion preference**
|
||||||
|
|
||||||
|
When reduce motion is on, VisualTimer shows static fill state instead of animated ring.
|
||||||
|
|
||||||
|
- [ ] **Step 5: Verify build**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd src-tauri && cargo check && cd .. && npm run build
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 6: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add src-tauri/ src/
|
||||||
|
git commit -m "feat(timer): wire VisualTimer to tasks with notifications and persistence"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 2C — Data & Polish
|
||||||
|
|
||||||
|
### Task 11: Export and Open Data
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `src/lib/utils/obsidianExport.js`
|
||||||
|
- Modify: `src/lib/pages/DictationPage.svelte`
|
||||||
|
- Modify: `src/lib/pages/HistoryPage.svelte`
|
||||||
|
- Modify: `src/lib/pages/TasksPage.svelte`
|
||||||
|
|
||||||
|
- [ ] **Step 1: Create obsidianExport.js**
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
export function exportTranscriptToObsidian(transcript, segments, tasks) {
|
||||||
|
const frontmatter = `---
|
||||||
|
title: "${transcript.title || 'Voice Note'}"
|
||||||
|
date: ${transcript.created_at}
|
||||||
|
source: ${transcript.source}
|
||||||
|
duration: ${transcript.duration}s
|
||||||
|
engine: ${transcript.engine}
|
||||||
|
tags: [kon, transcription]
|
||||||
|
---\n\n`;
|
||||||
|
// ... body with text + optional task list
|
||||||
|
}
|
||||||
|
|
||||||
|
export function exportTasksToJSON(tasks) { ... }
|
||||||
|
export function exportTasksToCSV(tasks) { ... }
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Add "Export to Obsidian" button to HistoryPage**
|
||||||
|
|
||||||
|
Uses `@tauri-apps/plugin-dialog` to pick output directory, then writes markdown files.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Add task export to TasksPage**
|
||||||
|
|
||||||
|
JSON and CSV export buttons.
|
||||||
|
|
||||||
|
- [ ] **Step 4: Verify build**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm run build
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 5: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add src/
|
||||||
|
git commit -m "feat(export): add Obsidian export, task JSON/CSV export"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 12: First Run Polish
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `src/lib/pages/FirstRunPage.svelte`
|
||||||
|
- Modify: `src/lib/stores/page.svelte.js`
|
||||||
|
|
||||||
|
- [ ] **Step 1: Add microphone permission request step**
|
||||||
|
|
||||||
|
Before model download, request mic permission via `navigator.mediaDevices.getUserMedia()`.
|
||||||
|
|
||||||
|
- [ ] **Step 2: Add test recording step**
|
||||||
|
|
||||||
|
After model loads, show a quick 5-second test recording: "Say something..." → display result → "You're ready!"
|
||||||
|
|
||||||
|
- [ ] **Step 3: Wire optional LLM download**
|
||||||
|
|
||||||
|
After STT model: "Want smarter task extraction? Download AI assistant ({size}, optional)"
|
||||||
|
|
||||||
|
- [ ] **Step 4: Time the flow — target under 90 seconds**
|
||||||
|
|
||||||
|
Add performance instrumentation to log total onboarding time.
|
||||||
|
|
||||||
|
- [ ] **Step 5: Verify build**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm run build
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 6: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add src/
|
||||||
|
git commit -m "feat(firstrun): add mic permission, test recording, LLM download step"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 13: Settings Wiring + Global Hotkey Update
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `src/lib/pages/SettingsPage.svelte`
|
||||||
|
- Modify: `src/lib/stores/page.svelte.js`
|
||||||
|
- Modify: `src/routes/+layout.svelte`
|
||||||
|
|
||||||
|
- [ ] **Step 1: Change default hotkey to Ctrl+Shift+Space**
|
||||||
|
|
||||||
|
In `page.svelte.js`, change `globalHotkey: "Ctrl+Shift+R"` to `globalHotkey: "Ctrl+Shift+Space"`.
|
||||||
|
|
||||||
|
- [ ] **Step 2: Add microphone selection setting**
|
||||||
|
|
||||||
|
Use `navigator.mediaDevices.enumerateDevices()` to list audio input devices. Display as dropdown in Settings. Pass selected device ID to AudioContext.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Wire export directory setting**
|
||||||
|
|
||||||
|
Use `@tauri-apps/plugin-dialog` for directory picker.
|
||||||
|
|
||||||
|
- [ ] **Step 4: Migrate remaining localStorage settings to preferences store**
|
||||||
|
|
||||||
|
The `settings` object in page.svelte.js currently uses localStorage. Add a `$effect` that syncs key settings to the SQLite-backed preferences store.
|
||||||
|
|
||||||
|
- [ ] **Step 5: Verify build**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm run build
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 6: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add src/
|
||||||
|
git commit -m "feat(settings): wire mic selection, export directory, update default hotkey"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 14: Final Validation
|
||||||
|
|
||||||
|
- [ ] **Step 1: Full build check**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm run build && cd src-tauri && cargo check
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Keyboard navigation**
|
||||||
|
|
||||||
|
Tab through every page. Verify focus rings visible.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Context restoration test**
|
||||||
|
|
||||||
|
Set non-default preferences → close app → relaunch. Verify state preserved.
|
||||||
|
|
||||||
|
- [ ] **Step 4: Reduce motion test**
|
||||||
|
|
||||||
|
Toggle reduce motion on → verify all animations stopped, timer shows static state.
|
||||||
|
|
||||||
|
- [ ] **Step 5: Commit any fixes**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add -A
|
||||||
|
git commit -m "fix(validation): final validation pass corrections"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
| Phase | Tasks | Key Deliverable |
|
||||||
|
|---|---|---|
|
||||||
|
| 2A: Core Pipeline (1–5) | Schema migration, transcript persistence, task CRUD, FTS5 search, frontend store migration | Working voice → text → SQLite pipeline |
|
||||||
|
| 2B: Intelligence (6–10) | LLM crate, model management, task extraction, micro-stepping, visual timer | AI-powered task decomposition with timer |
|
||||||
|
| 2C: Polish (11–14) | Export, first run, settings, validation | Ship-ready for closed beta |
|
||||||
|
|
||||||
|
**Total:** 14 tasks. Schema first. Backend commands before frontend. LLM after core pipeline works. Polish last.
|
||||||
|
|
||||||
|
**Critical path:** Task 1 (schema) → Task 2-3 (commands) → Task 4-5 (frontend migration) → Task 6-7 (LLM) → everything else.
|
||||||
|
|
||||||
|
**Risk:** llama-cpp-2 compilation on Windows requires MSVC + CMake. If it fails, Tasks 6-9 scope down to rule-based extraction only (already works).
|
||||||
Reference in New Issue
Block a user