Lumotia/docs/architecture-map/04-llm-formatting-mcp/README.md

---
name: Slice 4 — LLM, AI Formatting, MCP, Cloud Providers
type: architecture-map-page
slice: 04-llm-formatting-mcp
last_verified: 2026/05/09
---

# Slice 4: LLM, AI Formatting, MCP, Cloud Providers

> **Where you are:** Architecture map → LLM, Formatting, MCP

**Plain English summary.** This slice covers how Lumotia turns raw audio output into clean prose, surfaces the local LLM that powers cleanup and task extraction, exposes the user's transcripts to external agents over MCP, and reserves a stub crate for future bring-your-own-key cloud providers. Four crates, all on the inference / post-processing edge of the app.

## At a glance

- **Crates:** four. `lumotia-llm`, `lumotia-ai-formatting`, `lumotia-mcp` (binary + lib), `lumotia-cloud-providers`.
- **Total LOC:** ~3,520 (LLM 1,250, formatting 1,290, MCP 580, cloud-providers 80).
- **llama-cpp-2 version:** `0.1.144` (default features off, then `vulkan` and `openmp` re-enabled by Cargo features).
- **MCP protocol version:** `2024-11-05`, JSON-RPC 2.0 over stdio.
- **Model registry:** four-tier Qwen3.5 / Qwen3.6 family (2B / 4B / 9B / 27B, all Q4_K_M GGUF) with resumable HTTP download and SHA-256 verification.
- **Three high-level LLM surfaces:** `cleanup_text` (freeform), `decompose_task` (3–7 micro-steps under GBNF), `extract_tasks` (optional array under GBNF). Plus the Phase 9 `extract_content_tags` (one `{topic, intent}` pair under a closed-set GBNF).
- **Formatting pipeline stages:** anti-hallucination filter → filler removal → British English conversion → repetition collapse and basic capitalisation → smart paragraph breaks on long pauses → optional LLM cleanup with prompt-injection-hardened system prompt.
- **MCP tools:** `list_transcripts`, `get_transcript`, `search_transcripts`, `list_tasks`. Read-only at the SQLite connection layer.
- **Cloud providers:** scaffolding only. A process-local API key store with `MAGNOTIA_API_KEY_<PROVIDER>` env-var fallback. No transports, no STT calls.

## Map of this slice

LLM crate:

- [LLM engine and llama-cpp-2 lifecycle](llm-engine.md)
- [`cleanup_text` surface](llm-cleanup-text.md)
- [`decompose_task` surface](llm-decompose-task.md)
- [`extract_tasks` surface](llm-extract-tasks.md)
- [`extract_content_tags` surface](llm-extract-content-tags.md)
- [Prompts and grammars catalogue](llm-prompts-and-grammars.md)
- [Model manager (four-tier Qwen, downloads, SHA verification)](llm-model-manager.md)
- [Cargo features (gpu-vulkan, openmp)](llm-cargo-features.md)
- [Tests (smoke, content_tags_smoke, gating)](llm-tests.md)

AI formatting crate:

- [Pipeline overview](formatting-pipeline.md)
- [Filler removal and British English](formatting-filler-and-british.md)
- [Anti-hallucination filter](formatting-anti-hallucination.md)
- [Plain-text pre-formatter for LLM cleanup](formatting-plain-text-preformatter.md)
- [LLM cleanup bridge (`llm_client`)](formatting-llm-cleanup-bridge.md)
- [Correction learning (HITL → custom dictionary)](formatting-correction-learning.md)

MCP crate:

- [Server entry and stdio protocol](mcp-server.md)
- [Tools (list, get, search, list_tasks)](mcp-tools.md)

Cloud providers crate:

- [Stub crate state and BYOK plan](cloud-providers-stubs.md)

## How this slice connects to others

- **From slice 3 (audio + transcription):** transcription emits `Vec<Segment>` (a `lumotia-core::types::Segment`). `lumotia-ai-formatting::pipeline::post_process_segments` is the immediate consumer. Slice 4 has no compile-time dependency on the transcription crates; the contract is the `Segment` type alone.
- **To slice 4 internally:** `lumotia-ai-formatting` depends on `lumotia-llm`, never the other way. The LLM cleanup bridge (`llm_client`) lives in the formatting crate so the LLM crate stays free of post-processing concerns.
- **From slice 5 (core, storage, hotkey, build):** `lumotia-llm` consumes `lumotia_core::tuning::{inference_thread_count, Workload}` for thread sizing and `lumotia_core::paths::app_paths().llm_models_dir()` for the on-disk model store. `lumotia-ai-formatting` consumes `lumotia_core::types::Segment` and `lumotia_core::constants::SMART_PARAGRAPH_GAP_SECS`. `lumotia-mcp` opens `lumotia_storage::database_path()` via `lumotia_storage::init_readonly()` and calls `list_transcripts`, `get_transcript`, `search_transcripts`, `list_tasks` from `lumotia_storage`.
- **To slice 2 (Tauri runtime):** the Tauri commands at `src-tauri/src/commands/llm.rs`, `src-tauri/src/commands/tasks.rs`, `src-tauri/src/commands/transcription.rs`, `src-tauri/src/commands/live.rs`, `src-tauri/src/commands/profiles.rs`, and `src-tauri/src/commands/models.rs` are the only callers of this slice's public surfaces from inside the Tauri process. Each per-surface page lists its best-guess Tauri command for slice 2 reconciliation.

## Existing in-repo docs

- `docs/issues/llm-prompt-preflight.md` — RB-10 release-blocker write-up for the prompt token-budget preflight in `LlmEngine::generate`. Resolved 2026-04-22. Cross-referenced from [`llm-engine.md`](llm-engine.md).
- The 2026-04-22 code review is referenced from several MCP comments in `crates/mcp/src/lib.rs`. The review document lives at `docs/code-review-2026-04-22.md` (slice 5 territory).

## Open questions, debt, drift

- **Empty cloud-providers crate.** Only a process-local in-memory keystore is wired. No HTTP transports, no OpenAI / Anthropic clients, no STT calls. Documented in [`cloud-providers-stubs.md`](cloud-providers-stubs.md). The keystore TODO targets the `keyring` crate or platform-native credential storage.
- **Content-tags trivial-true cross-check observability gap.** `LlmEngine::generate` derives `gpu_offloaded` from `use_gpu && gpu_layers >= model.n_layer()`, which is trivially true today (`gpu_layers` is `u32::MAX` whenever `use_gpu` is set). True residency observability — parsing llama.cpp's "offloaded N/M layers" log line — is tracked in `docs/superpowers/specs/2026-05-09-battery-gpu-aware-thread-tuning-design.md` (§ Out of scope). Per commit `052265b`, the explicit comparison is left in place to document intent. See [`llm-engine.md`](llm-engine.md) for the call site.
- **Prompt versioning.** All system prompts and GBNF grammars live as `pub const &str` with no version tag. A change to `CLEANUP_PROMPT`, `DECOMPOSE_TASK_SYSTEM`, `EXTRACT_TASKS_SYSTEM`, or `CONTENT_TAGS_SYSTEM` is invisible to downstream callers and to any cached LLM output. Worth introducing a `PROMPT_VERSION` constant and stamping it onto persisted output once cached cleanup re-runs become a feature.
- **Model registry drift.** `LlmModelId::sha256()` and `LlmModelId::hf_url()` are pinned to specific Hugging Face revisions. Upstream re-uploads (rare but they happen) silently invalidate the SHA. There is no automated check that the registered URL still matches the registered SHA. Manual verification expected at upgrade time.
- **GBNF schema drift between content-tags GBNF and the closed set.** `INTENT_CLOSED_SET` (in `prompts.rs`) and the `intent` rule in `CONTENT_TAGS_GRAMMAR` (in `grammars.rs`) duplicate the same six values. They are kept in sync by hand. A drift would let the model emit a value that parses through the GBNF but fails `is_valid_intent` (the current code path) — or, worse, the other way round (GBNF blocks a value the closed set considers valid). A test that asserts the two stay in lock-step would close this gap.
- **Token-budget preflight is upper-bounded at the constant `MAX_CONTEXT_TOKENS = 8192`.** The 27B tier ships with a much larger native context but the engine never advertises it. Lifting the cap requires either per-model context windows in the registry or a probe of the loaded model's `n_ctx_train`. Tracked in [`llm-engine.md`](llm-engine.md).
- **MCP server has no auth, no transport-level scoping.** Stdio-only by design. Anyone with stdio access to the binary has read access to every transcript. Documented in [`mcp-server.md`](mcp-server.md). Cloud / HTTP transports would need an auth layer — out of scope today.