Lumotia/docs/architecture-map/04-llm-formatting-mcp/formatting-pipeline.md

name, type, slice, last_verified

name	type	slice	last_verified
AI formatting pipeline overview	architecture-map-page	04-llm-formatting-mcp	2026/05/09

AI formatting pipeline overview

Where you are: Architecture map → LLM, Formatting, MCP → Pipeline overview

Plain English summary. post_process_segments is the single entry point that takes a Vec<Segment> from transcription and applies the configured filters in order. It owns the order, the LLM gate, and the segment-collapse on LLM output. Every other module in this crate is a leaf the pipeline calls into.

At a glance

• Crate: lumotia-ai-formatting
• Path: crates/ai-formatting/src/pipeline.rs
• LOC: 211
• Public surface:
  • pub struct PostProcessOptions { remove_fillers, british_english, anti_hallucination, format_mode, dictionary_terms } (crates/ai-formatting/src/pipeline.rs:8)
  • pub enum FormatMode { Raw, Clean, Smart } (:20)
  • impl FormatMode { pub fn parse(&str) -> Self } (:27)
  • pub fn post_process_segments(segments: &mut Vec<Segment>, options: &PostProcessOptions, llm: Option<&LlmEngine>) (:38)
• External deps that matter: lumotia_core::constants::SMART_PARAGRAPH_GAP_SECS, lumotia_core::types::Segment, lumotia_llm::LlmEngine. Internal modules: llm_client, rule_based, to_plain_text.
• Tauri command that calls this (slice 2, best guess): three call sites, all in slice 2:
  • src-tauri/src/commands/transcription.rs:196, :317, :386 — the file-import and historical-transcript paths.
  • src-tauri/src/commands/live.rs:891 — the live dictation path.

What's in here

PostProcessOptions (crates/ai-formatting/src/pipeline.rs:8)

Bag of booleans plus a format mode plus a dictionary list. The struct is pub but not Clone or Default — callers always build it explicitly, which keeps "did I mean to enable this filter?" answered at the call site, not by accidentally inheriting a default.

Fields:

• remove_fillers: bool
• british_english: bool
• anti_hallucination: bool
• format_mode: FormatMode
• dictionary_terms: Vec<String> — per-user vocabulary. Forwarded into the LLM cleanup prompt so the model knows how to spell custom names. Documented inline in the struct.

FormatMode (crates/ai-formatting/src/pipeline.rs:20)

Three states with progressively more processing:

• Raw — only the segment-level filters (filler, British, anti-halluc) run. No formatting, no repetition collapse, no LLM.
• Clean — adds repetition collapse and basic capitalisation. No LLM by default; only invokes LLM if a loaded engine is supplied.
• Smart — Clean plus paragraph breaks on long pauses. Same LLM gate.

FormatMode::parse(s) accepts the strings the frontend serialises ("Clean", "Smart"); anything else falls back to Raw.

post_process_segments (crates/ai-formatting/src/pipeline.rs:38)

The pipeline. Steps in order:

1. Anti-hallucination filter (drop step). If options.anti_hallucination, retain only segments where rule_based::is_hallucination(&seg.text) returns false. This removes segments rather than rewriting them.
2. Per-segment rewrite loop. For each remaining segment:
   • If remove_fillers: seg.text = rule_based::remove_fillers(&seg.text).
   • If british_english: seg.text = rule_based::to_british_english(&seg.text).
   • If format_mode != Raw: seg.text = rule_based::collapse_repetitions(&seg.text) then seg.text = rule_based::format_text(&seg.text).
3. Smart paragraph breaks. If format_mode == Smart && segments.len() > 1, walk segments in reverse and prepend "\n\n" to any segment whose start - prev.end > SMART_PARAGRAPH_GAP_SECS. Reverse iteration avoids index drift.
4. Optional LLM cleanup. If llm: Some(&LlmEngine) is passed, the engine is loaded, and format_mode != Raw:
   • Pre-format the segments via to_plain_text(segments) — collapses to a single natural-language string with whitespace normalised and zero-width chars stripped.
   • If the joined string is non-empty, call llm_client::cleanup_text(engine, &joined, &options.dictionary_terms, LlmPromptPreset::Default).
   • On success with non-empty cleaned output: replace_segments_with_cleaned(segments, cleaned.trim()). The whole Vec<Segment> is replaced with a single Segment whose start is the original first segment's start, end is the original last segment's end, and text is the cleaned string.
   • On error: eprintln! the failure and keep the rule-based output. Cleanup never blocks the rule-based result. Stable degradation under model error.

The reason LlmPromptPreset::Default is hard-coded here: this entry point is the pipeline path used by file-imports and live transcribe. The "named preset" UX (Email, Notes, Code) goes through the explicit cleanup_transcript_text_cmd Tauri command, where the frontend supplies the preset. See formatting-llm-cleanup-bridge.md for the preset story.

replace_segments_with_cleaned (crates/ai-formatting/src/pipeline.rs:103)

Helper that does the segment collapse. Empty / blank cleaned strings short-circuit (no replace). The new single segment's timing covers the full original range so downstream consumers (timeline UIs, exports) still know where the audio sat.

Data flow

Vec<Segment>  +  PostProcessOptions  +  Option<&LlmEngine>
  → if anti_hallucination: retain(!is_hallucination(seg.text))
  → for each seg:
       remove_fillers (if enabled)
       to_british_english (if enabled)
       if format_mode != Raw:
         collapse_repetitions
         format_text
  → if format_mode == Smart and segments.len() > 1:
       walk reverse, prepend "\n\n" on long pauses
  → if llm and engine.is_loaded() and format_mode != Raw:
       joined = to_plain_text(segments)
       if !joined.is_empty():
         cleaned = llm_client::cleanup_text(engine, joined, dictionary_terms, Default)
         on Ok(cleaned) and non-empty:
           segments.clear(); push single Segment { start: first.start, end: last.end, text: cleaned }
         on Err: log and keep rule-based output
  → mutates segments in-place; no return

Watch-outs

• Order matters. Anti-hallucination runs first because some filler-removal patterns would corrupt a [blank_audio] marker into something the hallucination filter no longer recognises. Do not reorder without re-running the test suite — crates/ai-formatting/src/pipeline.rs:155-209 covers the expected interleaving.
• anti_hallucination is a drop, not a rewrite. A Segment filtered as a hallucination is gone from the output entirely. Tests at crates/ai-formatting/src/pipeline.rs:156-174 confirm this.
• format_mode == Raw skips the LLM, even if a loaded engine is supplied. This is the single switch users have for "just give me the rule-based result". Frontend gating depends on it.
• LLM cleanup collapses the segment list. A 50-segment transcript becomes one segment after a successful LLM call. Any downstream feature that assumes per-segment timing on the LLM-cleaned text needs to skip the LLM stage or run it after a fresh re-segmentation. Cleanup output's start and end cover the original range, but anything inside is opaque.
• LLM error path is silent (eprintln) but visible to the user. The rule-based output stays; the user sees rule-based text where they expected LLM-cleaned text. There is no surfacing back up to the Tauri layer beyond the eprintln. Worth a structured error if "did the LLM run" becomes user-visible.
• Dictionary terms only flow through the LLM path. The rule-based filters do not see dictionary_terms. A user-defined custom spelling that the rule-based BRITISH_REPLACEMENTS table contradicts will get re-Britished. The LLM cleanup prompt includes the terms explicitly to override that.
• SMART_PARAGRAPH_GAP_SECS lives in lumotia-core, not here. Slice 5 owns the constant. Look there to tune the long-pause threshold.

See also

• Filler removal and British English
• Anti-hallucination filter
• Plain-text pre-formatter
• LLM cleanup bridge
• Correction learning
• Slice README