Lumotia/docs/architecture-map/04-llm-formatting-mcp/README.md

name, type, slice, last_verified

name	type	slice	last_verified
Slice 4 — LLM, AI Formatting, MCP, Cloud Providers	architecture-map-page	04-llm-formatting-mcp	2026/05/09

Slice 4: LLM, AI Formatting, MCP, Cloud Providers

Where you are: Architecture map → LLM, Formatting, MCP

Plain English summary. This slice covers how Lumotia turns raw audio output into clean prose, surfaces the local LLM that powers cleanup and task extraction, exposes the user's transcripts to external agents over MCP, and reserves a stub crate for future bring-your-own-key cloud providers. Four crates, all on the inference / post-processing edge of the app.

At a glance

• Crates: four. lumotia-llm, lumotia-ai-formatting, lumotia-mcp (binary + lib), lumotia-cloud-providers.
• Total LOC: ~3,520 (LLM 1,250, formatting 1,290, MCP 580, cloud-providers 80).
• llama-cpp-2 version: 0.1.144 (default features off, then vulkan and openmp re-enabled by Cargo features).
• MCP protocol version: 2024-11-05, JSON-RPC 2.0 over stdio.
• Model registry: four-tier Qwen3.5 / Qwen3.6 family (2B / 4B / 9B / 27B, all Q4_K_M GGUF) with resumable HTTP download and SHA-256 verification.
• Three high-level LLM surfaces: cleanup_text (freeform), decompose_task (3–7 micro-steps under GBNF), extract_tasks (optional array under GBNF). Plus the Phase 9 extract_content_tags (one {topic, intent} pair under a closed-set GBNF).
• Formatting pipeline stages: anti-hallucination filter → filler removal → British English conversion → repetition collapse and basic capitalisation → smart paragraph breaks on long pauses → optional LLM cleanup with prompt-injection-hardened system prompt.
• MCP tools: list_transcripts, get_transcript, search_transcripts, list_tasks. Read-only at the SQLite connection layer.
• Cloud providers: scaffolding only. A process-local API key store with MAGNOTIA_API_KEY_<PROVIDER> env-var fallback. No transports, no STT calls.

Map of this slice

LLM crate:

• LLM engine and llama-cpp-2 lifecycle
• cleanup_text surface
• decompose_task surface
• extract_tasks surface
• extract_content_tags surface
• Prompts and grammars catalogue
• Model manager (four-tier Qwen, downloads, SHA verification)
• Cargo features (gpu-vulkan, openmp)
• Tests (smoke, content_tags_smoke, gating)

AI formatting crate:

• Pipeline overview
• Filler removal and British English
• Anti-hallucination filter
• Plain-text pre-formatter for LLM cleanup
• LLM cleanup bridge (llm_client)
• Correction learning (HITL → custom dictionary)

MCP crate:

• Server entry and stdio protocol
• Tools (list, get, search, list_tasks)

Cloud providers crate:

• Stub crate state and BYOK plan

How this slice connects to others

• From slice 3 (audio + transcription): transcription emits Vec<Segment> (a lumotia-core::types::Segment). lumotia-ai-formatting::pipeline::post_process_segments is the immediate consumer. Slice 4 has no compile-time dependency on the transcription crates; the contract is the Segment type alone.
• To slice 4 internally: lumotia-ai-formatting depends on lumotia-llm, never the other way. The LLM cleanup bridge (llm_client) lives in the formatting crate so the LLM crate stays free of post-processing concerns.
• From slice 5 (core, storage, hotkey, build): lumotia-llm consumes lumotia_core::tuning::{inference_thread_count, Workload} for thread sizing and lumotia_core::paths::app_paths().llm_models_dir() for the on-disk model store. lumotia-ai-formatting consumes lumotia_core::types::Segment and lumotia_core::constants::SMART_PARAGRAPH_GAP_SECS. lumotia-mcp opens lumotia_storage::database_path() via lumotia_storage::init_readonly() and calls list_transcripts, get_transcript, search_transcripts, list_tasks from lumotia_storage.
• To slice 2 (Tauri runtime): the Tauri commands at src-tauri/src/commands/llm.rs, src-tauri/src/commands/tasks.rs, src-tauri/src/commands/transcription.rs, src-tauri/src/commands/live.rs, src-tauri/src/commands/profiles.rs, and src-tauri/src/commands/models.rs are the only callers of this slice's public surfaces from inside the Tauri process. Each per-surface page lists its best-guess Tauri command for slice 2 reconciliation.

Existing in-repo docs

• docs/issues/llm-prompt-preflight.md — RB-10 release-blocker write-up for the prompt token-budget preflight in LlmEngine::generate. Resolved 2026-04-22. Cross-referenced from llm-engine.md.
• The 2026-04-22 code review is referenced from several MCP comments in crates/mcp/src/lib.rs. The review document lives at docs/code-review-2026-04-22.md (slice 5 territory).

Open questions, debt, drift

• Empty cloud-providers crate. Only a process-local in-memory keystore is wired. No HTTP transports, no OpenAI / Anthropic clients, no STT calls. Documented in cloud-providers-stubs.md. The keystore TODO targets the keyring crate or platform-native credential storage.
• Content-tags trivial-true cross-check observability gap. LlmEngine::generate derives gpu_offloaded from use_gpu && gpu_layers >= model.n_layer(), which is trivially true today (gpu_layers is u32::MAX whenever use_gpu is set). True residency observability — parsing llama.cpp's "offloaded N/M layers" log line — is tracked in docs/superpowers/specs/2026-05-09-battery-gpu-aware-thread-tuning-design.md (§ Out of scope). Per commit 052265b, the explicit comparison is left in place to document intent. See llm-engine.md for the call site.
• Prompt versioning. All system prompts and GBNF grammars live as pub const &str with no version tag. A change to CLEANUP_PROMPT, DECOMPOSE_TASK_SYSTEM, EXTRACT_TASKS_SYSTEM, or CONTENT_TAGS_SYSTEM is invisible to downstream callers and to any cached LLM output. Worth introducing a PROMPT_VERSION constant and stamping it onto persisted output once cached cleanup re-runs become a feature.
• Model registry drift. LlmModelId::sha256() and LlmModelId::hf_url() are pinned to specific Hugging Face revisions. Upstream re-uploads (rare but they happen) silently invalidate the SHA. There is no automated check that the registered URL still matches the registered SHA. Manual verification expected at upgrade time.
• GBNF schema drift between content-tags GBNF and the closed set. INTENT_CLOSED_SET (in prompts.rs) and the intent rule in CONTENT_TAGS_GRAMMAR (in grammars.rs) duplicate the same six values. They are kept in sync by hand. A drift would let the model emit a value that parses through the GBNF but fails is_valid_intent (the current code path) — or, worse, the other way round (GBNF blocks a value the closed set considers valid). A test that asserts the two stay in lock-step would close this gap.
• Token-budget preflight is upper-bounded at the constant MAX_CONTEXT_TOKENS = 8192. The 27B tier ships with a much larger native context but the engine never advertises it. Lifting the cap requires either per-model context windows in the registry or a probe of the loaded model's n_ctx_train. Tracked in llm-engine.md.
• MCP server has no auth, no transport-level scoping. Stdio-only by design. Anyone with stdio access to the binary has read access to every transcript. Documented in mcp-server.md. Cloud / HTTP transports would need an auth layer — out of scope today.