jake/Lumotia

Fork 0

Files

Jake 26c7307607

check / cargo check (macos-latest) (push) Has been cancelled

Details

check / cargo check (ubuntu-22.04) (push) Has been cancelled

Details

check / cargo check (windows-latest) (push) Has been cancelled

Details

check / svelte build + lint (push) Has been cancelled

Details

agent: lumotia-rebrand — docs, scripts, root config, residuals

Phase 9 of the rebrand cascade. Sweep covers everything the Phase 8
frontend pass deliberately skipped: docs/, root markdown, scripts,
Cargo.toml descriptions, code comments that survived earlier
word-boundary sed, plus a handful of identifiers caught on the final
verify pass.

transcription-app changes:
- README.md, HANDOVER.md, KNOWN-ISSUES.md, run.sh — magnotia/Magnotia
  -> lumotia/Lumotia.
- docs/ — sweep across all subdirs except docs/handovers/ (preserved
  as immutable audit trail). Includes architecture-map references
  to magnotia_core::*, magnotia_storage::*, etc. now pointing at
  lumotia_*; dev-setup.md tracing output examples (lumotia_startup
  target); brief/ + superpowers/ + issues/ + whisper-ecosystem/ +
  audit/.
- Cargo.toml descriptions on 9 crates (core, audio, cloud-providers,
  hotkey, llm, mcp, plus referenced others).
- crates/core/src/{error,hardware,recommendation,paths}.rs +
  crates/audio/src/wav.rs + crates/llm/src/model_manager.rs +
  crates/cloud-providers/src/keystore.rs + crates/mcp/src/lib.rs —
  doc comments and a model-manager user-agent string.
- Caught on final pass: BroadcastChannel("magnotia_task_sync") -> ...
  ("lumotia_task_sync"); magnotia_locale i18n localStorage key
  renamed + migration shim added; CSS keyframe names
  magnotiaPulse / magnotiaBar / magnotiaFade renamed in the design-
  system kit; magnotia_viewer_item / magnotia_viewer_mode handoff
  keys renamed in HistoryPage + viewer/+page.svelte; src/assets/
  wordmark.svg text.
- src-tauri/src/lib.rs comment cleanup ("magnotia era" was sed'd
  to "lumotia era" earlier — restored).

Preserved (intentional):
- crates/core/src/paths.rs — keeps "magnotia" / "Magnotia" / ".magnotia"
  legacy detection strings in legacy_and_target_paths() so the
  migration shim can still find user data from the magnotia era.
- src/lib/stores/{page,focusTimer}.svelte.ts + src/lib/i18n/index.ts
  — migration call sites reference the legacy magnotia keys
  deliberately.
- docs/handovers/ — historical audit trail.

cargo build --workspace passes. npm run check: 0 errors / 0 warnings
(3958 files). cargo test --workspace: 339 pass / 0 fail.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-13 12:38:03 +01:00

24 KiB

Raw Blame History

Lumotia — Whisper Ecosystem Research Brief

Spec handover for Claude Code. Mined across 10 open-source Whisper apps, synthesised April 2026. British English. Every claim URL-backed — see Sources.

TL;DR

Ten repos surveyed (whisper.cpp, whisper-rs, Handy, Buzz, Whispering/Epicenter, faster-whisper, WhisperLive, whisper_streaming, Scriberr, Vibe, plus OpenWhispr). The dominant pre-emptive patches for Lumotia cluster around silent CUDA fallback, global-hotkey edge cases, MSVC C-runtime linking between whisper-rs-sys and tokenizers, clipboard overwrite on paste, and Tauri HTTP scope blocking localhost LLM calls. The highest-leverage features to pinch are an engine-abstraction crate (transcribe-rs pattern), a named-prompt transformation pipeline with raw-transcript preservation, Vulkan as the default GPU path on Windows, VAD-gated chunking with a hallucination blocklist, and a warm-up audio file at launch.

Cross-repo pain pattern matrix

Pain point	Repos where observed	Severity for Lumotia	Recommended Lumotia mitigation
Silent CPU fallback when CUDA version mismatches	whisper.cpp #2857, #2214, #2297; faster-whisper #1398; Buzz FAQ; Handy #209	H	Detect actual compute device at runtime, log it, show in UI; prefer Vulkan default on Windows
Global-hotkey bugs (modifier-only, right-hand mods, F13–F24, single-key on Linux, Fn on macOS)	Handy #1143/#1105/#1019/#966/#956/#917/#47; Whispering #491/#500/#484/#549	H	Dedicated cross-platform hotkey layer; per-OS capability matrix; reject invalid combos in UI
`whisper-rs-sys` + `tokenizers` MSVC CRT conflict on Windows	Whispering v7.11.0	H	Avoid `tokenizers` crate in the same binary as whisper-rs on Windows; or route text pre/post-proc via an IPC'd sidecar
Audio-capture init race (first press records nothing, mic double-init)	Handy #1143, #1101, #1063	H	Warm CPAL stream at app start; debounce hotkey; serialise record/stop via state machine
Clipboard overwrite / not restored after paste	Handy #921 (fixed by PR #1040), #692 (direct-paste in terminals)	H	Snapshot clipboard, paste, restore on 200 ms timer on a background thread
Model download corrupted → crash on load	Buzz FAQ; Handy release notes	M	SHA checksum verify + resumable HTTP; keep raw audio on disk even if transcribe fails
Tauri `http://127.0.0.1:11434` blocked by scope allowlist	Vibe #438, #487	H	Pre-approve localhost LLM endpoint in `tauri.conf.json` capabilities; never surface the raw error
Ollama discoverability / localhost binding	Scriberr #111, #144; Vibe #440, #438	H	Auto-detect + "Test connection" button + prefilled defaults; visible status chip
VRAM contention between Whisper and LLM on one GPU	Scriberr #217	M	Pipeline: finish+unload Whisper, then load LLM; never run both concurrently by default
AVX2 / old-CPU `illegal instruction`	whisper-rs #8, #117; Buzz FAQ	M	Detect CPU flags at first launch; ship non-AVX2 fallback build
Chunk-boundary hallucinations ("Thanks for watching", "字幕by…")	WhisperLive #185, #246; ufal #121	H	Blocklist + `no_speech_prob`/`avg_logprob` gate + `condition_on_previous_text=false` on low-conf chunks
Buffer-bloat / latency cliff past 30 s	ufal #120, #102	H	Aggressive buffer trim tied to commit points, not wall clock
Prompt-loop poisoning (repetition cascades)	ufal #161	M	Detect repetition; reset context window; expose `repetition_penalty`
Cold-start first-chunk latency (~4–5 s)	ufal #96, #135	M	Run a silent warm-up WAV on launch + after idle
Wayland vs X11 / ALSA / Pipewire	Handy #1105, PRs #1025/#1042; Whispering (X11-only AppImage)	M	Target Pipewire first, gate Wayland global-shortcuts behind feature flag
macOS App Nap silently stops recording	Whispering #549, #559	M	Disable App Nap while recording (`NSProcessInfo` activity)
macOS code-sign / accessibility reprompt loops	Whispering PR #195, v4 hotfix	M	Notarise + entitlements; persist accessibility grant; document `xattr -cr` only as last resort
Windows DLL hell (libssl, vulkan-1.dll on CPU-only)	Buzz #1459; Whispering #840, #829	M	Bundle required DLLs in installer; fall back cleanly if Vulkan runtime absent
Direct-paste duplicates/breaks in terminals (Codex, Claude Code)	Handy #692	M	Detect terminal focus; switch paste mode to clipboard-only in those apps
Settings reset on update	Handy #602	L	Versioned schema migration; never destructive upgrade

Killer features inventory

Feature	Source repo(s)	Implementation complexity	Lumotia priority
Engine abstraction crate (Whisper / Parakeet / Moonshine behind one trait)	Handy, Whispering (both use `transcribe-rs`)	M	MVP — unblocks Windows fallback off whisper-rs
Vulkan backend as default Windows GPU path	whisper.cpp, Buzz, Vibe	S	MVP
GPU enumeration + auto-select UI with active-device indicator	Handy PR #1142	S	MVP
VAD-gated chunking with Silero	whisper.cpp, Buzz v1.4.4, faster-whisper	M	MVP (already in streaming scope)
Hallucination blocklist + confidence gate	WhisperLive #185, ufal #121	S	MVP
Warm-up WAV at launch	ufal (warm-up file PR)	S	MVP
LocalAgreement-n streaming commit policy	ufal	M	v1 (if Lumotia streaming currently naive)
Initial prompt / custom-vocab biasing	Whispering PR #1132, Buzz, Handy custom-words	S	MVP
Progressive audio save (crash-safe)	Whispering	S	MVP
Keep audio on disk when transcription fails, offer retry	Handy v0.8.0	S	MVP
Named prompt presets (email / notes / code / summary)	Scriberr, Whispering, OpenWhispr	S	MVP
Chained "transformation" pipeline (LLM → find/replace → LLM)	Whispering	M	v1
Dual LLM provider toggle (bundled local ↔ BYO cloud)	Vibe, OpenWhispr, Scriberr	S	v1
"Test connection" button for LLM endpoint	Vibe (pattern; buggy impl)	S	MVP
Raw-transcript always preserved; diff/undo LLM output	implicit in Scriberr, Whispering system prompt	S	MVP
Speaker-aware prompt substitution	Scriberr PR #294	S	v1
Plain-text (not JSON) fed to LLM	Scriberr PR #288	S	MVP
Streaming LLM output with cancel button	standard Ollama/llama.cpp	S	MVP
Sound cues on start/stop/complete	Handy	S	MVP
Pause-while-recording (mute other audio)	Handy PR #1028, #998	M	v1
Auto-start on system login	Whispering PR #1161	S	v1
Auto-update channel with safe rollback	Handy #883; Whispering	M	v1
Subtitle / SRT / VTT I/O	Buzz #1423/#1426	M	later
Folder-watcher batch mode	Scriberr, Buzz	M	later
MCP/CLI "run user command with STDIN/STDOUT" hook	Handy Discussion #211	M	later

Streaming-specific findings

VAD is necessary but not sufficient. Both WhisperLive and ufal ship Silero VAD and still produce "Thanks for watching!" / "字幕by…" artefacts at chunk edges. Layer a secondary defence: a phrase blocklist, a no_speech_prob / avg_logprob gate, and toggle condition_on_previous_text=false during low-confidence chunks.
Buffer management past the 30-second Whisper context is the top failure mode. ufal #120 and #102 show naive rolling buffers degrade catastrophically at long-session scale. Trim must be aggressive and tied to a confirmed commit point, not clock time.
LocalAgreement-n is the de-facto streaming policy for un-fine-tuned Whisper. Emission latency ≈ 2× chunk size, and every chunk is re-transcribed multiple times, which is costly but correct. Newer AlignAtt (ufal/SimulStreaming, IWSLT 2025 winner) is ~5× faster — flag for a later swap.
Prompt carry-over is a two-edged sword. Helpful for proper nouns (ufal's static_init_prompt), lethal on repetition loops (#161) or when the model wasn't trained with prompt conditioning (#133). Expose condition_on_previous_text as a runtime toggle and reset context on repetition detection.
Cold-start latency is universal (~4–5 s on first chunk). Run a short silent WAV at app launch and after any GPU/ANE idle.
Backend abstraction is mandatory for Lumotia's five-platform target. WhisperLive runs faster-whisper / TensorRT-LLM / OpenVINO behind one socket; ufal swaps faster-whisper / whisper-timestamped / MLX / OpenAI API. Bake this in early.
Reconnect/resume logic is under-engineered in both reference projects (WhisperLive #388). For mobile targets, design resumable streaming from day one — no good OSS reference exists.
First chunk triggers the full Whisper context window to allocate, causing the latency cliff. Pre-allocate on warm-up.
Multi-client fan-out (WhisperLive PR #174) is a clean pattern if Lumotia ever wants simultaneous transcribe + translate on one mic.

LLM formatting layer findings

Auto-apply LLM cleanup is where "LLM changed my meaning" complaints originate. None of Scriberr/Vibe/OpenWhispr have this class of issue in volume because they keep LLM post-processing explicitly opt-in. Lumotia auto-applies — make it one-keystroke revertable and always show the raw transcript.
Ollama connectivity is the #1 source of user-facing LLM bugs — Scriberr #111/#144, Vibe #438/#487. Solve with auto-detect, "Test connection", pre-approved Tauri HTTP capability for 127.0.0.1:11434 (and whichever port bundled llama.cpp uses), and a visible status chip.
VRAM contention (Scriberr #217): bundled LLM + Whisper on one consumer GPU evict each other. Default to sequential execution: finish Whisper, unload, run LLM.
Feed plain text to the LLM, not raw Whisper JSON with timestamps — Scriberr PR #288 made this switch and materially improved quality.
Substitute speaker labels ("SPEAKER_00" → user-assigned name) before prompting — Scriberr PR #294.
CUDA/cuDNN fragmentation pushes projects towards Vulkan — Vibe migrated off CUDA-exe (300 MB) to Vulkan whisper.cpp precisely to escape version hell. Same logic applies to llama.cpp — prefer Vulkan backend on Windows.
Non-English transcripts are already weaker (Vibe Vietnamese/Hebrew). LLM "cleanup" will compound errors. Always keep raw-transcript revert; never rewrite history in the paste buffer.
System-prompt framing matters. Whispering's published baseline — "translator from spoken to written form, not an editor trying to improve the content" — directly mitigates overcorrection. Copy this framing.
Settings per task (tiny model for titles, bigger for summaries) — Scriberr pattern. Lumotia can extend: faster model for paste-time cleanup, bigger for explicit summarise action.
Streaming LLM output with cancellation — standard Ollama/llama.cpp capability but rarely shipped; users hit X on 30-second summaries.

Atomic task backlog

Pre-emptive patches

Detect and surface active compute device. Pain: silent CUDA→CPU fallback. Accept: settings shows "GPU: Vulkan RTX 3060" or "CPU (fallback: driver 545 < required 555)".
Pre-approve http://127.0.0.1:* in Tauri capabilities for LLM endpoint. Pain: Vibe #438 opaque scope error. Accept: localhost LLM HTTP calls never hit scope error in any default install.
Add clipboard snapshot + restore-on-timer after paste. Pain: Handy #921. Accept: user's prior clipboard content is restored within 300 ms of paste.
Warm CPAL/WASAPI stream at app start; debounce hotkey trigger. Pain: Handy #1143 first-press-fails, #1101 double-init. Accept: first hotkey press post-launch captures audio from t=0.
Hotkey capability matrix per OS with UI rejection of invalid combos. Pain: Handy #966/#956/#917/#1019/#1105. Accept: user cannot bind single-key on X11, right-mod on Windows, or Fn on macOS; UI explains why.
Guard against whisper-rs-sys + tokenizers in same Windows binary. Pain: Whispering v7.11.0. Accept: Windows build either omits tokenizers or isolates text pre-processing in a sidecar process.
CPU feature detection at first launch with non-AVX2 fallback path. Pain: whisper-rs #8/#117. Accept: app starts without "illegal instruction" on a pre-2013 CPU.
Checksum-verify + resumable model downloads; retain audio when transcription fails. Pain: Buzz FAQ; Handy v0.8.0 pattern. Accept: corrupted download is re-fetched on next launch, raw WAV is kept for manual retry.
Disable macOS App Nap while recording and while LLM runs. Pain: Whispering #549. Accept: 10-minute background recording completes unattended.
Detect focused-app class; use clipboard-only paste in terminal emulators. Pain: Handy #692. Accept: typing into Windows Terminal, iTerm2, Kitty, Alacritty does not duplicate characters.
Versioned settings schema with forward-migration. Pain: Handy #602. Accept: settings from v0.x survive upgrade to v1.y without loss.
Bundle Vulkan loader and libssl DLLs in Windows installer with CPU-only fallback. Pain: Whispering #840/#829, Buzz #1459. Accept: app launches cleanly on a VM with no GPU and on a fresh Windows install.

Feature pinches

Introduce engine-abstraction trait (Transcriber) with Whisper + Parakeet backends. Pain/feature: Handy/Whispering transcribe-rs pattern; Windows whisper-rs CRT escape. Accept: engine swap at runtime via settings, no restart.
GPU enumeration and explicit device selector in settings. Feature: Handy PR #1142. Accept: dropdown lists all detected GPUs plus CPU; current active device highlighted.
Named LLM prompt presets (Email, Meeting Notes, Code, Quick Clean). Feature: Scriberr, Whispering. Accept: user picks preset from status-bar dropdown before dictating; prompts are user-editable.
System-prompt baseline framed as "translator, not editor". Feature: Whispering. Accept: default cleanup prompt is committed with that exact framing; overcorrection regression test passes.
Raw-transcript-always-preserved with one-keystroke revert. Feature: implicit in Scriberr/Whispering. Accept: after paste, ⌘/Ctrl+Z within 5 s replaces LLM output with raw transcript.
Initial-prompt / custom-vocab field for domain terms. Feature: Whispering PR #1132. Accept: user-supplied term list is passed as Whisper initial prompt and biases output.
Progressive audio write to disk during capture. Feature: Whispering. Accept: crash during transcription leaves a playable WAV in the session folder.
Sound cues for start / stop / complete, user-toggleable. Feature: Handy. Accept: three distinct short cues, volume slider, mute toggle.

Streaming

Silero-VAD-gated chunker with configurable threshold + hysteresis. Pain: WhisperLive #185. Accept: sustained background noise at 0.4–0.5 VAD score does not trigger transcription.
Hallucination blocklist + avg_logprob/no_speech_prob confidence gate. Pain: WhisperLive #185/#246, ufal #121. Accept: chunks containing only blocklisted phrases or below confidence threshold are dropped, not emitted.
Warm-up silent WAV on app launch and after 60 s idle. Pain: ufal #96/#135 cold-start. Accept: first user chunk post-warm-up completes in ≤ 1.5× steady-state latency.
LocalAgreement-n commit policy with configurable n. Feature: ufal. Accept: partial text is emitted only once confirmed across n=2 consecutive passes; tentative tail is visually distinct.
Aggressive buffer trim tied to commit points (not clock). Pain: ufal #120/#102. Accept: 10-minute continuous session does not exhibit latency growth past chunk 30.
Repetition detector that resets context window and drops the chunk. Pain: ufal #161. Accept: three consecutive identical tokens trigger context reset within one chunk.

LLM layer

"Test connection" button with proper error classification. Pain/feature: Vibe #438/#440. Accept: failure surfaces "Ollama not installed" / "port blocked" / "model not pulled" — never a raw URL scope error.
Sequential Whisper-then-LLM execution; shared GPU guard. Pain: Scriberr #217. Accept: on single-GPU systems, LLM does not load until Whisper has unloaded; configurable concurrent mode for users with ≥16 GB VRAM.
Plain-text pre-formatter before LLM prompt. Pain/feature: Scriberr PR #288. Accept: Whisper segments are joined into natural sentences before being sent to the LLM; timestamps stripped.
Streaming LLM output with cancel button. Feature: standard but underused. Accept: user can cancel mid-summary; partial output is kept or discarded per user pref.
Visible LLM status chip (disconnected / warming / generating). Pain: Ollama discoverability. Accept: chip reflects true state within 500 ms of change.

What couldn't be verified

GitHub /issues?q=...sort=reactions-%2B1-desc URLs were blocked at the fetch layer for several repos. Ordering of feature requests by raw reaction count is therefore inferred from comment volume, maintainer engagement, and release-note inclusion rather than numeric reaction tallies.
whisper-rs (GitHub) was archived on 30 July 2025; development continues at codeberg.org/tazz4843/whisper-rs. Issue numbers cited post that date will not resolve on GitHub. Lumotia should track Codeberg or fork.
OpenWhispr issue tracker was not fully mined within budget — included as a named reference for the bundled-llama.cpp architecture only. Pain-point claims for it are sourced from its README and a third-party dev.to writeup, not from its issues.
Whispering / EpicenterHQ issue tracker was not fully mined — the repo was cited via its release notes and README; its issue base may contain further pain patterns worth a follow-up.
Scriberr, Vibe PR numbers (#288, #294, #438, #487, #217, #111, #144) were sourced via search-result snippets where direct issue-page fetches were refused; titles and content are quoted but exact current status (open/closed/merged as of 21 Apr 2026) was not re-checked on each.
faster-whisper is named MIT but its heavy Python + CUDA runtime footprint makes it a pattern source, not a recommended Lumotia dependency.
Scriberr is MIT, not AGPL as the brief had hedged — full lifting is permitted; the "reference only" warning in the original task spec does not apply.
WhisperLive is MIT confirmed — full lifting permitted.
ufal/whisper_streaming is maintenance-only; upstream momentum has moved to ufal/SimulStreaming (also MIT). Consider tracking that repo for v1+.
Mobile (iOS, Android) pain surface is under-represented in this survey — none of the ten desktop repos have a mature mobile story beyond whisper.cpp's XCFramework and WhisperLive's ios-client. Lumotia's mobile backlog will need a dedicated pass.

Sources

24 KiB Raw Blame History Unescape Escape

Lumotia — Whisper Ecosystem Research Brief

TL;DR

Cross-repo pain pattern matrix

Killer features inventory

Streaming-specific findings

LLM formatting layer findings

Atomic task backlog

Pre-emptive patches

Feature pinches

Streaming

LLM layer

What couldn't be verified

Sources

24 KiB

Raw Blame History