Lumotia/docs/architecture-map/04-llm-formatting-mcp/formatting-plain-text-preformatter.md

---
name: Plain-text pre-formatter for LLM cleanup
type: architecture-map-page
slice: 04-llm-formatting-mcp
last_verified: 2026/05/09
---

# Plain-text pre-formatter for LLM cleanup

> **Where you are:** [Architecture map](../README.md) → [LLM, Formatting, MCP](README.md) → Plain-text pre-formatter

**Plain English summary.** Before the formatting pipeline calls the LLM, it joins all the segments into a single natural-language string with timestamps stripped and whitespace normalised. Per-segment structure is dropped because LLM cleanup quality degrades materially when fed timestamped JSON. Empty and zero-width-only segments are filtered out.

## At a glance

- Crate: `lumotia-ai-formatting`
- Path: `crates/ai-formatting/src/to_plain_text.rs`
- LOC: 223
- Public surface: `pub fn to_plain_text(segments: &[Segment]) -> String` (`crates/ai-formatting/src/to_plain_text.rs:33`)
- External deps that matter: `lumotia_core::types::Segment`. Pure CPU.
- Tauri command that calls this (slice 2, best guess): not called directly. Reaches Tauri via `pipeline::post_process_segments` (`crates/ai-formatting/src/pipeline.rs:76`).

## What's in here

### Provenance

The module-level doc-comment cites its source: brief item #29, sourced from Scriberr PR #288. Feeding raw Whisper JSON (with per-segment timestamps and structure) degraded LLM cleanup quality measurably; plain-text input raised it back. Lumotia's `Segment.text` field already holds just the spoken text — the timing lives in `start: f64` and `end: f64` — so "timestamp stripping" falls out of using the text field alone. The work here is the whitespace pass and empty-segment filter.

### `to_plain_text` (`crates/ai-formatting/src/to_plain_text.rs:33`)

Steps:

1. For each `Segment`: take `text`, run `normalise_whitespace`, then `trim`.
2. Drop empty results.
3. Join the survivors with a single ASCII space.
4. Run `normalise_whitespace` once more on the joined result so segment-boundary whitespace does not produce double spaces.
5. Final `trim` on the result.

Returns an empty string if every segment filtered out. No panics.

### `normalise_whitespace` (private, `crates/ai-formatting/src/to_plain_text.rs:56`)

Single-pass walk. For each char:

- **Zero-width format chars** (`is_zero_width_format`, `:86`): `U+200B`, `U+200C`, `U+200D`, `U+2060`, `U+FEFF`. Stripped without emitting anything. The `prev_was_space` flag is *not* updated, so a zero-width char between two spaces still collapses correctly to a single space.
- **Whitespace** (Unicode `is_whitespace()`): emit a single ASCII space, then suppress further consecutive whitespace until a non-space char arrives.
- **Anything else**: emit verbatim, reset the suppression flag.

Why zero-widths are stripped rather than collapsed: they are not whitespace in the Unicode sense (their `is_whitespace()` returns false), but they carry no natural-language content. Letting them through to the LLM wastes tokens and can confuse tokenisation. Treating them as "delete entirely" rather than "collapse to a space" avoids silently inserting word breaks where the source had none.

Codepoints covered:

- `U+200B ZERO WIDTH SPACE`
- `U+200C ZERO WIDTH NON-JOINER`
- `U+200D ZERO WIDTH JOINER`
- `U+2060 WORD JOINER`
- `U+FEFF ZERO WIDTH NO-BREAK SPACE` (also BOM)

## Data flow

```
&[Segment]
  → for each Segment:
       segment.text → normalise_whitespace → trim → keep if non-empty
  → join with " "
  → normalise_whitespace (idempotent re-pass)
  → trim
  → return String
```

## Tests

Comprehensive test suite at `crates/ai-formatting/src/to_plain_text.rs:93`:

- `empty_input_is_empty_output` (`:106`)
- `single_segment_returns_its_text_trimmed` (`:111`)
- `multiple_segments_are_joined_with_single_space` (`:117`)
- `empty_and_whitespace_segments_are_filtered` (`:123`) — covers `""`, `"   "`, `"\n\t "` mixed in with real segments.
- `internal_whitespace_runs_collapse_to_single_space` (`:135`) — within a segment.
- `join_boundary_does_not_produce_double_spaces` (`:141`) — the second-pass `normalise_whitespace` test.
- `non_breaking_space_is_treated_as_whitespace` (`:148`) — `U+00A0`. Unicode `is_whitespace` returns true here, so collapse is correct.
- `zero_width_format_chars_strip_entirely` (`:157`) — all five codepoints.
- `zero_width_chars_do_not_break_adjacent_whitespace_collapsing` (`:179`) — `"hello \u{FEFF} world"` collapses correctly.
- `leading_bom_is_stripped` (`:187`) — common artefact when Whisper reads a file with a BOM.
- `newlines_inside_segments_collapse` (`:195`) — `"line one\nline two\n\nline three"` → `"line one line two line three"`.
- `idempotent_on_already_normalised_text` (`:201`) — second call does not mangle.
- `only_empty_segments_yields_empty_string` (`:210`).
- `no_panic_on_pathological_whitespace_runs` (`:216`) — 10,000-space stress test.

## Watch-outs

- **Newlines collapse to spaces.** The pre-formatter is for the LLM cleanup prompt; it deliberately removes paragraph structure because the cleanup prompt is supposed to re-impose structure based on content, not legacy segmentation. Anything that actually needs paragraph breaks must use the pipeline's smart-pause logic before the LLM stage.
- **Idempotency is asserted by test, not by structure.** A second call to `to_plain_text` on the output of the first must produce the same string. Tests cover this; if a future change adds a transformation that is not idempotent, the test will catch it.
- **`is_whitespace` is the Unicode definition.** That includes NBSP (`\u{00A0}`), em space, and the rest of the family. All collapse to ASCII space.
- **Zero-width set is closed by the function.** Adding a new "invisible" codepoint requires updating `is_zero_width_format`. The standard Unicode "default ignorable" property would catch more codepoints but is not used here — the explicit allowlist keeps behaviour predictable.
- **No `trim_matches` on segment-level output before the second `normalise_whitespace`.** A segment that ends with a newline gets normalised to a trailing space, which the join then handles; the second `normalise_whitespace` collapses it. Working as designed but worth knowing if profiling ever flags the double-pass.
- **Output is one string. Caller (the pipeline) replaces the entire segment list with a single segment when this string then gets cleaned by the LLM.** Per-segment timing is not preservable through `to_plain_text`. This is by design — see [`formatting-pipeline.md`](formatting-pipeline.md) for the segment-collapse step.

## See also

- [Pipeline overview](formatting-pipeline.md)
- [LLM cleanup bridge](formatting-llm-cleanup-bridge.md)
- [Slice README](README.md)