--- name: AI formatting pipeline overview type: architecture-map-page slice: 04-llm-formatting-mcp last_verified: 2026/05/09 --- # AI formatting pipeline overview > **Where you are:** [Architecture map](../README.md) → [LLM, Formatting, MCP](README.md) → Pipeline overview **Plain English summary.** `post_process_segments` is the single entry point that takes a `Vec` from transcription and applies the configured filters in order. It owns the order, the LLM gate, and the segment-collapse on LLM output. Every other module in this crate is a leaf the pipeline calls into. ## At a glance - Crate: `lumotia-ai-formatting` - Path: `crates/ai-formatting/src/pipeline.rs` - LOC: 211 - Public surface: - `pub struct PostProcessOptions { remove_fillers, british_english, anti_hallucination, format_mode, dictionary_terms }` (`crates/ai-formatting/src/pipeline.rs:8`) - `pub enum FormatMode { Raw, Clean, Smart }` (`:20`) - `impl FormatMode { pub fn parse(&str) -> Self }` (`:27`) - `pub fn post_process_segments(segments: &mut Vec, options: &PostProcessOptions, llm: Option<&LlmEngine>)` (`:38`) - External deps that matter: `lumotia_core::constants::SMART_PARAGRAPH_GAP_SECS`, `lumotia_core::types::Segment`, `lumotia_llm::LlmEngine`. Internal modules: `llm_client`, `rule_based`, `to_plain_text`. - Tauri command that calls this (slice 2, best guess): three call sites, all in slice 2: - `src-tauri/src/commands/transcription.rs:196`, `:317`, `:386` — the file-import and historical-transcript paths. - `src-tauri/src/commands/live.rs:891` — the live dictation path. ## What's in here ### `PostProcessOptions` (`crates/ai-formatting/src/pipeline.rs:8`) Bag of booleans plus a format mode plus a dictionary list. The struct is `pub` but not `Clone` or `Default` — callers always build it explicitly, which keeps "did I mean to enable this filter?" answered at the call site, not by accidentally inheriting a default. Fields: - `remove_fillers: bool` - `british_english: bool` - `anti_hallucination: bool` - `format_mode: FormatMode` - `dictionary_terms: Vec` — per-user vocabulary. Forwarded into the LLM cleanup prompt so the model knows how to spell custom names. Documented inline in the struct. ### `FormatMode` (`crates/ai-formatting/src/pipeline.rs:20`) Three states with progressively more processing: - `Raw` — only the segment-level filters (filler, British, anti-halluc) run. No formatting, no repetition collapse, no LLM. - `Clean` — adds repetition collapse and basic capitalisation. No LLM by default; only invokes LLM if a loaded engine is supplied. - `Smart` — `Clean` plus paragraph breaks on long pauses. Same LLM gate. `FormatMode::parse(s)` accepts the strings the frontend serialises (`"Clean"`, `"Smart"`); anything else falls back to `Raw`. ### `post_process_segments` (`crates/ai-formatting/src/pipeline.rs:38`) The pipeline. Steps in order: 1. **Anti-hallucination filter (drop step).** If `options.anti_hallucination`, retain only segments where `rule_based::is_hallucination(&seg.text)` returns false. This *removes* segments rather than rewriting them. 2. **Per-segment rewrite loop.** For each remaining segment: - If `remove_fillers`: `seg.text = rule_based::remove_fillers(&seg.text)`. - If `british_english`: `seg.text = rule_based::to_british_english(&seg.text)`. - If `format_mode != Raw`: `seg.text = rule_based::collapse_repetitions(&seg.text)` then `seg.text = rule_based::format_text(&seg.text)`. 3. **Smart paragraph breaks.** If `format_mode == Smart && segments.len() > 1`, walk `segments` in reverse and prepend `"\n\n"` to any segment whose `start - prev.end > SMART_PARAGRAPH_GAP_SECS`. Reverse iteration avoids index drift. 4. **Optional LLM cleanup.** If `llm: Some(&LlmEngine)` is passed, the engine is loaded, and `format_mode != Raw`: - Pre-format the segments via `to_plain_text(segments)` — collapses to a single natural-language string with whitespace normalised and zero-width chars stripped. - If the joined string is non-empty, call `llm_client::cleanup_text(engine, &joined, &options.dictionary_terms, LlmPromptPreset::Default)`. - On success with non-empty cleaned output: `replace_segments_with_cleaned(segments, cleaned.trim())`. The whole `Vec` is replaced with a single `Segment` whose `start` is the original first segment's start, `end` is the original last segment's end, and `text` is the cleaned string. - On error: `eprintln!` the failure and keep the rule-based output. Cleanup never blocks the rule-based result. Stable degradation under model error. The reason `LlmPromptPreset::Default` is hard-coded here: this entry point is the pipeline path used by file-imports and live transcribe. The "named preset" UX (Email, Notes, Code) goes through the explicit `cleanup_transcript_text_cmd` Tauri command, where the frontend supplies the preset. See [`formatting-llm-cleanup-bridge.md`](formatting-llm-cleanup-bridge.md) for the preset story. ### `replace_segments_with_cleaned` (`crates/ai-formatting/src/pipeline.rs:103`) Helper that does the segment collapse. Empty / blank cleaned strings short-circuit (no replace). The new single segment's timing covers the full original range so downstream consumers (timeline UIs, exports) still know where the audio sat. ## Data flow ``` Vec + PostProcessOptions + Option<&LlmEngine> → if anti_hallucination: retain(!is_hallucination(seg.text)) → for each seg: remove_fillers (if enabled) to_british_english (if enabled) if format_mode != Raw: collapse_repetitions format_text → if format_mode == Smart and segments.len() > 1: walk reverse, prepend "\n\n" on long pauses → if llm and engine.is_loaded() and format_mode != Raw: joined = to_plain_text(segments) if !joined.is_empty(): cleaned = llm_client::cleanup_text(engine, joined, dictionary_terms, Default) on Ok(cleaned) and non-empty: segments.clear(); push single Segment { start: first.start, end: last.end, text: cleaned } on Err: log and keep rule-based output → mutates segments in-place; no return ``` ## Watch-outs - **Order matters.** Anti-hallucination runs first because some filler-removal patterns would corrupt a `[blank_audio]` marker into something the hallucination filter no longer recognises. Do not reorder without re-running the test suite — `crates/ai-formatting/src/pipeline.rs:155-209` covers the expected interleaving. - **`anti_hallucination` is a drop, not a rewrite.** A `Segment` filtered as a hallucination is gone from the output entirely. Tests at `crates/ai-formatting/src/pipeline.rs:156-174` confirm this. - **`format_mode == Raw` skips the LLM, even if a loaded engine is supplied.** This is the single switch users have for "just give me the rule-based result". Frontend gating depends on it. - **LLM cleanup collapses the segment list.** A 50-segment transcript becomes one segment after a successful LLM call. Any downstream feature that assumes per-segment timing on the LLM-cleaned text needs to skip the LLM stage or run it after a fresh re-segmentation. Cleanup output's `start` and `end` cover the original range, but anything inside is opaque. - **LLM error path is silent (eprintln) but visible to the user.** The rule-based output stays; the user sees rule-based text where they expected LLM-cleaned text. There is no surfacing back up to the Tauri layer beyond the eprintln. Worth a structured error if "did the LLM run" becomes user-visible. - **Dictionary terms only flow through the LLM path.** The rule-based filters do not see `dictionary_terms`. A user-defined custom spelling that the rule-based BRITISH_REPLACEMENTS table contradicts will get re-Britished. The LLM cleanup prompt includes the terms explicitly to override that. - **`SMART_PARAGRAPH_GAP_SECS` lives in `lumotia-core`, not here.** Slice 5 owns the constant. Look there to tune the long-pause threshold. ## See also - [Filler removal and British English](formatting-filler-and-british.md) - [Anti-hallucination filter](formatting-anti-hallucination.md) - [Plain-text pre-formatter](formatting-plain-text-preformatter.md) - [LLM cleanup bridge](formatting-llm-cleanup-bridge.md) - [Correction learning](formatting-correction-learning.md) - [Slice README](README.md)