Lumotia/README.md

# Magnotia

*Think out loud. Keep working.*

Magnotia is a local-first, cognitive-load-aware dictation and task-capture desktop app. Every transcription, LLM cleanup, and task extraction runs on the user's machine. No telemetry, no analytics, no cloud dependency. The app is designed around a single observation: people who think in bursts lose ideas faster than they can type, and the tool's job is to get out of the way.

---

## Status

**Pre-alpha.** Actively dogfooded on Linux (KDE Plasma 6 on Wayland). macOS and Windows targets are in scope and exercised by CI, but not yet beta-ready. One primary user; open source-intent with licence TBD before public beta.

- Current `main`: see commit log
- 9 library crates plus the Tauri app crate; 220+ lib tests plus 67 Tauri-app tests, all passing
- Cross-platform CI (Linux / macOS / Windows) via GitHub Actions
- Tracked limitations live in [`KNOWN-ISSUES.md`](KNOWN-ISSUES.md)

---

## Design principles (non-negotiable)

1. **Local-first is the floor, not a feature.** No voice, transcript, or task ever leaves the user's machine unless they explicitly send it. No telemetry.
2. **Cognitive load is the limiting resource.** Every new setting must earn its mental real estate. Every interaction should reduce, not add, decisions.
3. **Composable, not monolithic.** Magnotia is a dictation primitive: via MCP, CLI, and filesystem export, it slots into whatever workflow the user already has (Obsidian, Claude Desktop, Cline, any text field).
4. **LLM scope is narrow.** The in-app LLM does transcription cleanup and task extraction. It is not a wake-word agent, not a chat UI, not a multi-provider cloud fan-out.
5. **Raw transcript is always recoverable.** Cleanup is additive, never destructive. The user can always see and revert to what Whisper heard.

These are enforced in the codebase (where practical) and in the docs under [`docs/whisper-ecosystem/magnotia-context.md`](docs/whisper-ecosystem/magnotia-context.md).

---

## What Magnotia does today

### Speech-to-text
- Vulkan-accelerated local **Whisper** inference via [whisper-rs](https://github.com/tazz4843/whisper-rs) 0.16 + whisper.cpp. Works on NVIDIA, AMD, Intel Arc, Apple (via MoltenVK), and integrated graphics.
- Vulkan / CUDA-accelerated **Parakeet** inference via sherpa-onnx (NVIDIA's English-only model; lower latency than Whisper-Large on English).
- **Six Whisper variants** shipped: Tiny, Base, Small, Distil-Small, Medium, Distil-Large v3.
- **Parakeet-as-default for English** when hardware supports it; first-run hardware probe picks the fastest-accurate pair.
- **Resumable downloads with SHA-256 verification**; retains audio if transcription fails.
- **Per-profile custom vocabulary** fed to Whisper as `initial_prompt` plus to the LLM cleanup prompt; bulk import via paste.
- **Live streaming transcription** with speech-gated chunking, hallucination filtering, and duplicate-boundary detection.

### LLM formatting (local only)
- Local LLM runtime via [llama-cpp-2](https://github.com/utilityai/llama-cpp-rs) 0.1.144 with Vulkan.
- Four Qwen tiers (Qwen3.5 2B / 4B / 9B + Qwen3.6 27B) auto-selected by hardware probe.
- GBNF grammar-constrained output for task extraction (always-parseable JSON).
- System prompt hardened against voice-delivered prompt injection.

### Task capture
- Automatic task extraction from any transcript.
- **MicroSteps** — one-tap "break this task into 3–7 concrete physical actions."
- Profile-scoped task lists with inbox / today / soon / later buckets.
- Tasks back-link to their source transcript.

### Input, paste, and window management
- **Global hotkey** — evdev-based on Linux (Wayland-compatible out of the box), `tauri-plugin-global-shortcut` on macOS / Windows. Per-OS capability matrix rejects invalid key combinations.
- **Platform-aware paste matrix** — `wtype` / `xdotool` / `ydotool` on Linux, AppleScript on macOS, SendKeys on Windows. Clipboard snapshot + 300 ms restore after paste.
- **Wayland-hardened transcription preview overlay** (`/preview`): pinned across virtual desktops, hidden from Alt+Tab via `WindowTypeHint::Utility`, never steals focus, focus-gated open.
- **Meeting auto-capture** (opt-in, default off): single-signal process-list watcher, user-editable app list, surfaces a non-modal reminder. No mic-activity heuristics, no calendar integration.

### History and search
- **FTS5-indexed transcript search** over SQLite.
- **YAML-frontmatter markdown export** one-click into Obsidian vault.
- Per-transcript metadata: starred, manual tags, template, language, duration.
- Transcript editor window (`/viewer`) with debounced autosave.

### External integration
- **MCP stdio server** (`magnotia-mcp`) exposing read-only transcripts and tasks to any Model Context Protocol client (Claude Desktop, Cline, Cursor, etc.). No authentication, read-only, local-only.

### Accessibility
- Dyslexia-friendly fonts bundled: Lexend, Atkinson Hyperlegible Next, OpenDyslexic.
- Bionic reading mode.
- Per-region font size, letter spacing, line height, transcript-specific sizing.
- System-aware reduce-motion.
- **i18n**: English, Spanish, German (svelte-i18n scaffold).

### Privacy, deployment, reliability
- Zero telemetry. Zero analytics. No crash reports leave the machine unless explicitly bundled.
- Auto-update via Tauri updater plugin (signed, user-approved).
- Per-window size + position persistence (`tauri-plugin-window-state`).
- Crash + panic capture stored locally; user-bundleable for support.

---

## Architecture

Magnotia is a Tauri 2 desktop app with three layers:

```
┌─────────────────────────────────────────────────────────────────┐
│  Svelte 5 frontend (src/)                                       │
│  Routes: /, /float, /viewer, /preview                           │
│  Stores, i18n, Tailwind CSS                                     │
├─────────────────────────────────────────────────────────────────┤
│  Tauri 2 runtime (src-tauri/)                                   │
│  Commands: audio, clipboard, diagnostics, feedback, fs,         │
│            hardware, hotkey, intentions, live, llm, meeting,    │
│            models, nudges, paste, profiles, rituals, tasks,     │
│            transcription, transcripts, tts, update, windows     │
│  Utility modules (no commands): mod, power, security            │
│  Plugins: global-shortcut, dialog, opener, updater,             │
│           window-state                                          │
├─────────────────────────────────────────────────────────────────┤
│  Rust workspace (crates/)                                       │
│  magnotia-core, magnotia-audio, magnotia-transcription, magnotia-llm,               │
│  magnotia-ai-formatting, magnotia-storage, magnotia-hotkey,                    │
│  magnotia-cloud-providers, magnotia-mcp                                   │
└─────────────────────────────────────────────────────────────────┘
```

The Rust workspace is the brain; Tauri is the OS integration surface; Svelte is the UI. The MCP server (`magnotia-mcp`) is a separate binary that opens Magnotia's SQLite store read-only — it's Magnotia-as-primitive for external agents.

### Repository layout

```
magnotia/
├── Cargo.toml              # workspace root
├── src-tauri/              # Tauri app (main binary + commands)
│   ├── src/
│   │   ├── commands/       # 22 Tauri command modules + 3 utility modules (`mod`, `power`, `security`)
│   │   ├── lib.rs          # app entry, setup, command registration
│   │   ├── tray.rs
│   │   └── main.rs
│   ├── capabilities/       # Tauri ACL capability files
│   ├── gen/schemas/        # auto-generated ACL schemas
│   ├── tauri.conf.json     # base Tauri config
│   ├── tauri.linux.conf.json  # Linux overlay (native decorations)
│   └── resources/windows/  # Windows-specific bundled assets
├── crates/                 # workspace Rust crates
│   ├── ai-formatting/      # post-processing pipeline + LLM cleanup client
│   ├── audio/              # capture, resampling, decoding, WAV I/O
│   ├── cloud-providers/    # BYOK cloud STT stubs (empty scaffolding)
│   ├── core/               # types, hardware probe, model registry, process watch
│   ├── hotkey/             # Linux evdev hotkey listener
│   ├── llm/                # llama-cpp-2 engine + model manager
│   ├── mcp/                # MCP stdio server binary
│   ├── storage/            # SQLite + FTS5 + file storage
│   └── transcription/      # Whisper + Parakeet wrappers, model mgmt
├── src/                    # Svelte frontend
│   ├── routes/             # SvelteKit routes
│   │   ├── +page.svelte    # main dictation UI
│   │   ├── +layout.svelte  # shell (sidebar, tray sync, hotkey wiring)
│   │   ├── float/          # tasks float window
│   │   ├── viewer/         # transcript editor window
│   │   └── preview/        # transcription preview overlay
│   ├── lib/
│   │   ├── pages/          # DictationPage, SettingsPage, HistoryPage, TasksPage, FilesPage, FirstRunPage
│   │   ├── components/     # reusable Svelte components
│   │   ├── stores/         # $state stores (page, preferences, profiles, toasts)
│   │   ├── actions/        # Svelte actions (bionic reading, etc.)
│   │   ├── utils/          # frontmatter, textMeasure, errors, storage helpers
│   │   ├── types/          # TS type definitions
│   │   └── i18n/           # svelte-i18n setup + en/es/de locales
│   ├── fonts/              # bundled accessibility fonts
│   ├── design-system/      # design tokens + UI kit references (not live code)
│   └── app.css
├── docs/                   # all project documentation (see below)
├── .github/workflows/      # CI (check.yml, build.yml)
├── package.json
├── HANDOVER.md             # latest session handover
└── run.sh                  # dev launcher (starts Vite then Tauri)
```

### Rust crates

| Crate | Responsibility |
|---|---|
| **`magnotia-core`** | Shared types (`Segment`, `Transcript`, `Megabytes`, `ModelId`), constants, the `Engine` / `SpeedTier` / `AccuracyTier` enums, hardware probe (`sysinfo`-based), model registry (Whisper + Parakeet entries), hardware-aware recommendation scoring, `process_watch` for meeting detection. |
| **`magnotia-audio`** | `cpal`-based microphone capture with device hotplug + error forwarding, VAD, `rubato` streaming resampler to 16 kHz mono, `symphonia` file decoding, `hound` WAV I/O. |
| **`magnotia-transcription`** | `whisper-rs` backend (`WhisperRsBackend`) that owns a `WhisperContext` and supports `set_initial_prompt`. `LocalEngine` wraps both Whisper and Parakeet (via `transcribe-rs` ONNX) behind a common `Transcriber` trait. Streaming primitives (`VadChunker`, `LocalAgreement`, buffer trim) live in the `streaming/` module. Model manager handles downloads, paths, and disk checks. |
| **`magnotia-llm`** | `llama-cpp-2` engine with a four-tier Qwen3.5 / Qwen3.6 model manager. Three high-level surfaces: `cleanup_text` (formatting), `decompose_task` (3–7 micro-steps, GBNF-constrained JSON array), `extract_tasks` (optional-array, GBNF-constrained). Resumable HTTP downloads with SHA-256 verify. |
| **`magnotia-ai-formatting`** | Post-processing pipeline: filler removal, British English conversion, anti-hallucination filter, smart paragraph breaks on long pauses, optional LLM cleanup. Also hosts the `llm_client::CLEANUP_PROMPT` constant (prompt-injection-hardened). |
| **`magnotia-storage`** | SQLite via `sqlx` 0.8. Migrations, CRUD for transcripts / tasks / subtasks / profiles / profile terms / settings / error log, FTS5 search, file-storage paths. |
| **`magnotia-hotkey`** | Linux `evdev` hotkey listener with device hotplug. Parses Tauri-style hotkey strings (`Ctrl+Shift+R`), emits Pressed / Released events. Works natively on Wayland (no X11 dependency). Checks `/dev/input/event*` access on startup; surfaces a clear "add yourself to the `input` group" error when missing. |
| **`magnotia-cloud-providers`** | BYOK cloud-STT provider stubs. Currently empty scaffolding. When populated: OpenAI-compatible endpoint + Anthropic (ceiling for scope). |
| **`magnotia-mcp`** | Standalone `magnotia-mcp` binary implementing the MCP stdio protocol (2024-11-05). Read-only tools: `list_transcripts`, `get_transcript`, `search_transcripts`, `list_tasks`. Opens Magnotia's SQLite store. |

### Tauri commands (src-tauri/src/commands/)

| Module | What it exposes |
|---|---|
| `audio` | Device enumeration, native capture start/stop, audio-samples persistence |
| `clipboard` | Cross-platform clipboard write (arboard) |
| `diagnostics` | Panic hook, frontend error log, crash file listing, diagnostic report bundler |
| `feedback` | Thumbs / correction capture on AI-generated output; few-shot example store for prompt conditioning |
| `fs` | Thin filesystem write for the OS save-dialog path (UTF-8 text, dialog-constrained) |
| `hardware` | `probe_system`, `rank_models` |
| `hotkey` | `start_evdev_hotkey`, `update_evdev_hotkey`, `stop_evdev_hotkey`, `check_hotkey_access`, `is_wayland_session` |
| `intentions` | Implementation-intention rule CRUD (if-then automation: time-of-day, task-completed, morning-triage triggers) |
| `live` | Live streaming transcription session lifecycle + speech-gate tuning |
| `llm` | Tier recommend, model check / download / load / unload / delete, status, `cleanup_transcript_text_cmd`, `extract_tasks_from_transcript_cmd` |
| `meeting` | `detect_meeting_processes` (process-list poll) |
| `models` | Whisper + Parakeet model download / load / check / default-id resolution, runtime capabilities API, pre-warm |
| `nudges` | Margot soft-touch nudge delivery via `tauri-plugin-notification`; main-window-only guard |
| `paste` | `paste_text` (copy + keystroke), `detect_paste_backends`, Wayland focus-race mitigation against the preview overlay |
| `profiles` | Profile CRUD, profile-terms CRUD, learn-terms-from-edit |
| `rituals` | Start- and shutdown-ritual sentinels (last-shown date for the morning-triage modal) |
| `tasks` | Task CRUD, subtask CRUD, `decompose_and_store`, `extract_tasks_from_transcript_cmd` |
| `transcription` | `transcribe_pcm`, `transcribe_file`, `transcribe_pcm_parakeet` |
| `transcripts` | Transcript CRUD + FTS5 search |
| `tts` | Platform-native Read Page Aloud (`spd-say` / `say` / PowerShell), with cancellable child-process tracking |
| `update` | Tauri-plugin-updater check / install |
| `windows` | `open_task_window`, `open_viewer_window`, `open_preview_window`, `close_preview_window` |

Utility modules in the same directory (no `#[tauri::command]` attributes; helpers consumed by the command modules above): `mod` (registry), `power` (macOS `PowerAssertion` guard against App Nap during long sessions), `security` (`ensure_main_window` guard).

### Frontend (src/)

- **SvelteKit + Svelte 5 runes** (`$state`, `$derived`, `$effect`).
- **Tailwind CSS 4** for styling, with a Lexend/Atkinson/OpenDyslexic type system.
- **Secondary windows** (`/float`, `/viewer`, `/preview`) use named layouts (`+layout@.svelte`) to skip the main shell and run chrome-free.
- **Reactive stores** (`src/lib/stores/`, one file per store): `page.svelte.ts` (central app state; transcripts, profiles, taskLists, templates, etc. live as fields here), `preferences.svelte.ts`, `profiles.svelte.ts`, `toasts.svelte.ts`, `focusTimer.svelte.ts`, `llmStatus.svelte.ts`, `nudgeBus.svelte.ts`, `implementationIntentions.svelte.ts`, `completionStats.svelte.ts`, `speaker.svelte.ts`.
- **i18n**: `svelte-i18n` with en/es/de locales at `src/lib/i18n/locales/`. Scaffolding only — strings migrate to translation keys incrementally.

---

## Runtime stack

| Layer | Technology | Version |
|---|---|---|
| Desktop framework | [Tauri](https://tauri.app) | 2.10.3 |
| Frontend | Svelte 5 + SvelteKit + Vite | latest |
| Styling | Tailwind CSS | 4.x |
| Speech-to-text (primary) | whisper.cpp via [`whisper-rs`](https://github.com/tazz4843/whisper-rs) | 0.16 (Vulkan feature) |
| Speech-to-text (Parakeet) | sherpa-onnx via `transcribe-rs` | 0.3 |
| Local LLM | [`llama-cpp-2`](https://github.com/utilityai/llama-cpp-rs) | 0.1.144 (openmp + vulkan) |
| Database | SQLite via [`sqlx`](https://github.com/launchbadge/sqlx) | 0.8 |
| Async runtime | [`tokio`](https://tokio.rs/) | 1.x |
| Audio capture | [`cpal`](https://github.com/RustAudio/cpal) | current |
| Resampling | [`rubato`](https://github.com/HEnquist/rubato) | current |
| File decode | [`symphonia`](https://github.com/pdeljanov/Symphonia) | current |

---

## Platform support

| Platform | Status | Notes |
|---|---|---|
| Linux Wayland (KDE Plasma, GNOME Mutter, Hyprland, Sway) | **Primary target**, daily-dogfooded on KDE | evdev hotkey, GTK 3 via webkit2gtk, Vulkan, all paste backends; idle inhibit not wired (see KI-02) |
| Linux X11 | Supported | xdotool paste path, GTK 3; idle inhibit not wired (see KI-02) |
| macOS | In CI, untested runtime | osascript paste, Metal via MoltenVK, App Nap guard pending Apple Silicon verification (see KI-01) |
| Windows | In CI, untested runtime | SendKeys paste, Vulkan-first GPU path, bundled DLLs for CPU fallback; sleep prevention not wired (see KI-03) |

CI runs `cargo check --workspace --all-targets` + `svelte-check` on all three on every push and PR.

---

## Build + development

### Prerequisites

Linux (Fedora/RHEL listed; adjust for your distro):
```
sudo dnf install libclang-devel clang \
    webkit2gtk4.1-devel libappindicator-gtk3-devel librsvg2-devel \
    alsa-lib-devel systemd-devel cmake \
    vulkan-headers vulkan-loader-devel glslc
```

macOS:
```
brew install cmake llvm vulkan-headers vulkan-loader molten-vk shaderc
```

Windows:
```
choco install cmake llvm vulkan-sdk
```

See [`docs/dev-setup.md`](docs/dev-setup.md) for the authoritative per-platform dependency list and for how `LIBCLANG_PATH` should be set.

### Dev launch

Canonical full-stack dev launch — starts Vite, waits for port 1420, then launches Tauri:

```bash
npm run dev:tauri
```

Direct shell equivalent:

```bash
./run.sh
```

For pure frontend iteration without Tauri:

```bash
npm run dev:frontend
```

### Build

```bash
npm run tauri build    # release build, produces .AppImage / .deb / .dmg / .msi / .exe
```

CI also builds release installers on tag push (see `.github/workflows/build.yml`).

### Testing

```bash
cargo test --workspace --lib    # 220+ lib tests across 9 library crates
npm run check                   # svelte-check (type-checks .svelte files)
cargo check --workspace --all-targets
```

---

## Project documentation

Beyond this README, the repo ships extensive internal documentation:

### Product + strategy — `docs/brief/`
Research briefs, competitive analysis, and strategic framing. Start with:
- [`what-magnotia-is.md`](docs/brief/what-magnotia-is.md) — product thesis
- [`why-current-tools-fail.md`](docs/brief/why-current-tools-fail.md) — market gap
- [`design-principles.md`](docs/brief/design-principles.md) — full principle list
- [`target-audience.md`](docs/brief/target-audience.md), [`market-size-demographics.md`](docs/brief/market-size-demographics.md)
- Appendices on cognitive ergonomics, AI body doubling, evolutionary psychology, implementation intentions, HITL scaffolding, voice interfaces

### Brand — `docs/brand/`
- [`magnotia-brand-guidelines.md`](docs/brand/magnotia-brand-guidelines.md)
- [`magnotia-brand-platform.md`](docs/brand/magnotia-brand-platform.md)

### Technical research — `docs/whisper-ecosystem/`
Cross-repo survey of 10 OSS Whisper projects, the Magnotia-specific atomic task backlog, and the two Cursor workstream plans.
- [`brief.md`](docs/whisper-ecosystem/brief.md) — 31-item task backlog (the canonical research spec)
- [`magnotia-context.md`](docs/whisper-ecosystem/magnotia-context.md) — ideology, shipped state, file-ownership fence for cloud AI agents
- [`workstream-A.md`](docs/whisper-ecosystem/workstream-A.md), [`workstream-B.md`](docs/whisper-ecosystem/workstream-B.md) — executed workstream plans

### GPU tuning — `docs/gpu-tuning/`
- [`plan.md`](docs/gpu-tuning/plan.md) — MVP plan for GGML env-var panel + `magnotia-bench` auto-tuner + `magnotia-configs` community repo

### Session handovers
- [`HANDOVER.md`](HANDOVER.md) — latest session summary
- Dated historical handovers under [`docs/handovers/`](docs/handovers/): `HANDOVER-2026-04-17.md`, `HANDOVER-2026-04-18.md`, `HANDOVER-2026-04-19.md`, `HANDOVER-2026-04-24.md`

### Dev reference
- [`docs/dev-setup.md`](docs/dev-setup.md) — dependency + launch reference
- [`docs/icon-mapping.md`](docs/icon-mapping.md) — icon conventions
- [`KNOWN-ISSUES.md`](KNOWN-ISSUES.md) — tracked partial implementations and limitations

---

## Roadmap

The shipped code represents Phases 1–3 and a partial Phase 4.

Pinned roadmap items (scoped in docs and session memory):

- **Phase 4** — remaining items from [`workstream-A.md`](docs/whisper-ecosystem/workstream-A.md) + [`workstream-B.md`](docs/whisper-ecosystem/workstream-B.md)
- **Voice calibration** — three-tier plan replacing the hardcoded speech-gate with per-user baselines
- **GPU community tuning** — see [`docs/gpu-tuning/plan.md`](docs/gpu-tuning/plan.md); five-phase roadmap from settings panel to agentic auto-tuner + community config repo
- **Cloud endpoint contract test** — when `magnotia-cloud-providers` grows a real provider
- **`ggml` dedup** — replace the interim `-Wl,--allow-multiple-definition` link flag with a proper shared-lib setup; unblocks custom shader / backend work
- **Mobile (iOS / Android)** — long-horizon, gated on the single-binary Rust stack scaling

Explicitly shelved (not coming without specific community signal):
- Wake-word / always-listening agent
- Chat-style LLM UI
- Multi-provider cloud fan-out beyond OpenAI-compatible + Anthropic
- Second notes-editing surface (transcripts leave Magnotia via frontmatter to Obsidian)
- Speaker diarization
- Dragon-style passage-based speaker fine-tuning (Whisper has no speaker adaptation)

---

## Contributing

Pre-alpha status; contribution process TBD before public beta. For now:

- Every Tauri command change must register in both [`src-tauri/src/lib.rs`](src-tauri/src/lib.rs) (invoke handler) and in the invoking frontend code.
- Every Settings-visible setting must have a type field in [`src/lib/types/app.ts`](src/lib/types/app.ts) and a default in [`src/lib/stores/page.svelte.ts`](src/lib/stores/page.svelte.ts).
- Every new workspace crate needs a `description` in its `Cargo.toml`.
- Tests: add at least a smoke test per new Tauri command or crate module. The workspace test floor is "no regressions on main."
- Wayland compatibility is a first-class concern — don't assume X11. The preview overlay and paste matrix live-document what this looks like in practice.

---

## Licence

To be finalised before public beta. Current intent: MIT or similar permissive licence, with Corbel Consulting offering optional commercial support / managed services as the revenue path.

---

## Contact

**Jake Sames** — [jakeadriansames@gmail.com](mailto:jakeadriansames@gmail.com)
Repo: [github.com/jakejars/magnotia](https://github.com/jakejars/magnotia) · [git.corbel.consulting/jake/magnotia](https://git.corbel.consulting/jake/magnotia)