Lumotia/docs/architecture-map/04-llm-formatting-mcp/formatting-pipeline.md

---
name: AI formatting pipeline overview
type: architecture-map-page
slice: 04-llm-formatting-mcp
last_verified: 2026/05/09
---

# AI formatting pipeline overview

> **Where you are:** [Architecture map](../README.md) → [LLM, Formatting, MCP](README.md) → Pipeline overview

**Plain English summary.** `post_process_segments` is the single entry point that takes a `Vec<Segment>` from transcription and applies the configured filters in order. It owns the order, the LLM gate, and the segment-collapse on LLM output. Every other module in this crate is a leaf the pipeline calls into.

## At a glance

- Crate: `lumotia-ai-formatting`
- Path: `crates/ai-formatting/src/pipeline.rs`
- LOC: 211
- Public surface:
  - `pub struct PostProcessOptions { remove_fillers, british_english, anti_hallucination, format_mode, dictionary_terms }` (`crates/ai-formatting/src/pipeline.rs:8`)
  - `pub enum FormatMode { Raw, Clean, Smart }` (`:20`)
  - `impl FormatMode { pub fn parse(&str) -> Self }` (`:27`)
  - `pub fn post_process_segments(segments: &mut Vec<Segment>, options: &PostProcessOptions, llm: Option<&LlmEngine>)` (`:38`)
- External deps that matter: `lumotia_core::constants::SMART_PARAGRAPH_GAP_SECS`, `lumotia_core::types::Segment`, `lumotia_llm::LlmEngine`. Internal modules: `llm_client`, `rule_based`, `to_plain_text`.
- Tauri command that calls this (slice 2, best guess): three call sites, all in slice 2:
  - `src-tauri/src/commands/transcription.rs:196`, `:317`, `:386` — the file-import and historical-transcript paths.
  - `src-tauri/src/commands/live.rs:891` — the live dictation path.

## What's in here

### `PostProcessOptions` (`crates/ai-formatting/src/pipeline.rs:8`)

Bag of booleans plus a format mode plus a dictionary list. The struct is `pub` but not `Clone` or `Default` — callers always build it explicitly, which keeps "did I mean to enable this filter?" answered at the call site, not by accidentally inheriting a default.

Fields:

- `remove_fillers: bool`
- `british_english: bool`
- `anti_hallucination: bool`
- `format_mode: FormatMode`
- `dictionary_terms: Vec<String>` — per-user vocabulary. Forwarded into the LLM cleanup prompt so the model knows how to spell custom names. Documented inline in the struct.

### `FormatMode` (`crates/ai-formatting/src/pipeline.rs:20`)

Three states with progressively more processing:

- `Raw` — only the segment-level filters (filler, British, anti-halluc) run. No formatting, no repetition collapse, no LLM.
- `Clean` — adds repetition collapse and basic capitalisation. No LLM by default; only invokes LLM if a loaded engine is supplied.
- `Smart` — `Clean` plus paragraph breaks on long pauses. Same LLM gate.

`FormatMode::parse(s)` accepts the strings the frontend serialises (`"Clean"`, `"Smart"`); anything else falls back to `Raw`.

### `post_process_segments` (`crates/ai-formatting/src/pipeline.rs:38`)

The pipeline. Steps in order:

1. **Anti-hallucination filter (drop step).** If `options.anti_hallucination`, retain only segments where `rule_based::is_hallucination(&seg.text)` returns false. This *removes* segments rather than rewriting them.
2. **Per-segment rewrite loop.** For each remaining segment:
   - If `remove_fillers`: `seg.text = rule_based::remove_fillers(&seg.text)`.
   - If `british_english`: `seg.text = rule_based::to_british_english(&seg.text)`.
   - If `format_mode != Raw`: `seg.text = rule_based::collapse_repetitions(&seg.text)` then `seg.text = rule_based::format_text(&seg.text)`.
3. **Smart paragraph breaks.** If `format_mode == Smart && segments.len() > 1`, walk `segments` in reverse and prepend `"\n\n"` to any segment whose `start - prev.end > SMART_PARAGRAPH_GAP_SECS`. Reverse iteration avoids index drift.
4. **Optional LLM cleanup.** If `llm: Some(&LlmEngine)` is passed, the engine is loaded, and `format_mode != Raw`:
   - Pre-format the segments via `to_plain_text(segments)` — collapses to a single natural-language string with whitespace normalised and zero-width chars stripped.
   - If the joined string is non-empty, call `llm_client::cleanup_text(engine, &joined, &options.dictionary_terms, LlmPromptPreset::Default)`.
   - On success with non-empty cleaned output: `replace_segments_with_cleaned(segments, cleaned.trim())`. The whole `Vec<Segment>` is replaced with a single `Segment` whose `start` is the original first segment's start, `end` is the original last segment's end, and `text` is the cleaned string.
   - On error: `eprintln!` the failure and keep the rule-based output. Cleanup never blocks the rule-based result. Stable degradation under model error.

The reason `LlmPromptPreset::Default` is hard-coded here: this entry point is the pipeline path used by file-imports and live transcribe. The "named preset" UX (Email, Notes, Code) goes through the explicit `cleanup_transcript_text_cmd` Tauri command, where the frontend supplies the preset. See [`formatting-llm-cleanup-bridge.md`](formatting-llm-cleanup-bridge.md) for the preset story.

### `replace_segments_with_cleaned` (`crates/ai-formatting/src/pipeline.rs:103`)

Helper that does the segment collapse. Empty / blank cleaned strings short-circuit (no replace). The new single segment's timing covers the full original range so downstream consumers (timeline UIs, exports) still know where the audio sat.

## Data flow

```
Vec<Segment>  +  PostProcessOptions  +  Option<&LlmEngine>
  → if anti_hallucination: retain(!is_hallucination(seg.text))
  → for each seg:
       remove_fillers (if enabled)
       to_british_english (if enabled)
       if format_mode != Raw:
         collapse_repetitions
         format_text
  → if format_mode == Smart and segments.len() > 1:
       walk reverse, prepend "\n\n" on long pauses
  → if llm and engine.is_loaded() and format_mode != Raw:
       joined = to_plain_text(segments)
       if !joined.is_empty():
         cleaned = llm_client::cleanup_text(engine, joined, dictionary_terms, Default)
         on Ok(cleaned) and non-empty:
           segments.clear(); push single Segment { start: first.start, end: last.end, text: cleaned }
         on Err: log and keep rule-based output
  → mutates segments in-place; no return
```

## Watch-outs

- **Order matters.** Anti-hallucination runs first because some filler-removal patterns would corrupt a `[blank_audio]` marker into something the hallucination filter no longer recognises. Do not reorder without re-running the test suite — `crates/ai-formatting/src/pipeline.rs:155-209` covers the expected interleaving.
- **`anti_hallucination` is a drop, not a rewrite.** A `Segment` filtered as a hallucination is gone from the output entirely. Tests at `crates/ai-formatting/src/pipeline.rs:156-174` confirm this.
- **`format_mode == Raw` skips the LLM, even if a loaded engine is supplied.** This is the single switch users have for "just give me the rule-based result". Frontend gating depends on it.
- **LLM cleanup collapses the segment list.** A 50-segment transcript becomes one segment after a successful LLM call. Any downstream feature that assumes per-segment timing on the LLM-cleaned text needs to skip the LLM stage or run it after a fresh re-segmentation. Cleanup output's `start` and `end` cover the original range, but anything inside is opaque.
- **LLM error path is silent (eprintln) but visible to the user.** The rule-based output stays; the user sees rule-based text where they expected LLM-cleaned text. There is no surfacing back up to the Tauri layer beyond the eprintln. Worth a structured error if "did the LLM run" becomes user-visible.
- **Dictionary terms only flow through the LLM path.** The rule-based filters do not see `dictionary_terms`. A user-defined custom spelling that the rule-based BRITISH_REPLACEMENTS table contradicts will get re-Britished. The LLM cleanup prompt includes the terms explicitly to override that.
- **`SMART_PARAGRAPH_GAP_SECS` lives in `lumotia-core`, not here.** Slice 5 owns the constant. Look there to tune the long-pause threshold.

## See also

- [Filler removal and British English](formatting-filler-and-british.md)
- [Anti-hallucination filter](formatting-anti-hallucination.md)
- [Plain-text pre-formatter](formatting-plain-text-preformatter.md)
- [LLM cleanup bridge](formatting-llm-cleanup-bridge.md)
- [Correction learning](formatting-correction-learning.md)
- [Slice README](README.md)