Lumotia/docs/architecture-map/04-llm-formatting-mcp/formatting-plain-text-preformatter.md

name, type, slice, last_verified

name	type	slice	last_verified
Plain-text pre-formatter for LLM cleanup	architecture-map-page	04-llm-formatting-mcp	2026/05/09

Plain-text pre-formatter for LLM cleanup

Where you are: Architecture map → LLM, Formatting, MCP → Plain-text pre-formatter

Plain English summary. Before the formatting pipeline calls the LLM, it joins all the segments into a single natural-language string with timestamps stripped and whitespace normalised. Per-segment structure is dropped because LLM cleanup quality degrades materially when fed timestamped JSON. Empty and zero-width-only segments are filtered out.

At a glance

• Crate: lumotia-ai-formatting
• Path: crates/ai-formatting/src/to_plain_text.rs
• LOC: 223
• Public surface: pub fn to_plain_text(segments: &[Segment]) -> String (crates/ai-formatting/src/to_plain_text.rs:33)
• External deps that matter: lumotia_core::types::Segment. Pure CPU.
• Tauri command that calls this (slice 2, best guess): not called directly. Reaches Tauri via pipeline::post_process_segments (crates/ai-formatting/src/pipeline.rs:76).

What's in here

Provenance

The module-level doc-comment cites its source: brief item #29, sourced from Scriberr PR #288. Feeding raw Whisper JSON (with per-segment timestamps and structure) degraded LLM cleanup quality measurably; plain-text input raised it back. Lumotia's Segment.text field already holds just the spoken text — the timing lives in start: f64 and end: f64 — so "timestamp stripping" falls out of using the text field alone. The work here is the whitespace pass and empty-segment filter.

to_plain_text (crates/ai-formatting/src/to_plain_text.rs:33)

Steps:

1. For each Segment: take text, run normalise_whitespace, then trim.
2. Drop empty results.
3. Join the survivors with a single ASCII space.
4. Run normalise_whitespace once more on the joined result so segment-boundary whitespace does not produce double spaces.
5. Final trim on the result.

Returns an empty string if every segment filtered out. No panics.

normalise_whitespace (private, crates/ai-formatting/src/to_plain_text.rs:56)

Single-pass walk. For each char:

• Zero-width format chars (is_zero_width_format, :86): U+200B, U+200C, U+200D, U+2060, U+FEFF. Stripped without emitting anything. The prev_was_space flag is not updated, so a zero-width char between two spaces still collapses correctly to a single space.
• Whitespace (Unicode is_whitespace()): emit a single ASCII space, then suppress further consecutive whitespace until a non-space char arrives.
• Anything else: emit verbatim, reset the suppression flag.

Why zero-widths are stripped rather than collapsed: they are not whitespace in the Unicode sense (their is_whitespace() returns false), but they carry no natural-language content. Letting them through to the LLM wastes tokens and can confuse tokenisation. Treating them as "delete entirely" rather than "collapse to a space" avoids silently inserting word breaks where the source had none.

Codepoints covered:

• U+200B ZERO WIDTH SPACE
• U+200C ZERO WIDTH NON-JOINER
• U+200D ZERO WIDTH JOINER
• U+2060 WORD JOINER
• U+FEFF ZERO WIDTH NO-BREAK SPACE (also BOM)

Data flow

&[Segment]
  → for each Segment:
       segment.text → normalise_whitespace → trim → keep if non-empty
  → join with " "
  → normalise_whitespace (idempotent re-pass)
  → trim
  → return String

Tests

Comprehensive test suite at crates/ai-formatting/src/to_plain_text.rs:93:

• empty_input_is_empty_output (:106)
• single_segment_returns_its_text_trimmed (:111)
• multiple_segments_are_joined_with_single_space (:117)
• empty_and_whitespace_segments_are_filtered (:123) — covers "", " ", "\n\t " mixed in with real segments.
• internal_whitespace_runs_collapse_to_single_space (:135) — within a segment.
• join_boundary_does_not_produce_double_spaces (:141) — the second-pass normalise_whitespace test.
• non_breaking_space_is_treated_as_whitespace (:148) — U+00A0. Unicode is_whitespace returns true here, so collapse is correct.
• zero_width_format_chars_strip_entirely (:157) — all five codepoints.
• zero_width_chars_do_not_break_adjacent_whitespace_collapsing (:179) — "hello \u{FEFF} world" collapses correctly.
• leading_bom_is_stripped (:187) — common artefact when Whisper reads a file with a BOM.
• newlines_inside_segments_collapse (:195) — "line one\nline two\n\nline three" → "line one line two line three".
• idempotent_on_already_normalised_text (:201) — second call does not mangle.
• only_empty_segments_yields_empty_string (:210).
• no_panic_on_pathological_whitespace_runs (:216) — 10,000-space stress test.

Watch-outs

• Newlines collapse to spaces. The pre-formatter is for the LLM cleanup prompt; it deliberately removes paragraph structure because the cleanup prompt is supposed to re-impose structure based on content, not legacy segmentation. Anything that actually needs paragraph breaks must use the pipeline's smart-pause logic before the LLM stage.
• Idempotency is asserted by test, not by structure. A second call to to_plain_text on the output of the first must produce the same string. Tests cover this; if a future change adds a transformation that is not idempotent, the test will catch it.
• is_whitespace is the Unicode definition. That includes NBSP (\u{00A0}), em space, and the rest of the family. All collapse to ASCII space.
• Zero-width set is closed by the function. Adding a new "invisible" codepoint requires updating is_zero_width_format. The standard Unicode "default ignorable" property would catch more codepoints but is not used here — the explicit allowlist keeps behaviour predictable.
• No trim_matches on segment-level output before the second normalise_whitespace. A segment that ends with a newline gets normalised to a trailing space, which the join then handles; the second normalise_whitespace collapses it. Working as designed but worth knowing if profiling ever flags the double-pass.
• Output is one string. Caller (the pipeline) replaces the entire segment list with a single segment when this string then gets cleaned by the LLM. Per-segment timing is not preservable through to_plain_text. This is by design — see formatting-pipeline.md for the segment-collapse step.

See also

• Pipeline overview
• LLM cleanup bridge
• Slice README