Lumotia/docs/architecture-map/04-llm-formatting-mcp/formatting-filler-and-british.md

---
name: Filler removal, British English, repetition collapse, basic formatting
type: architecture-map-page
slice: 04-llm-formatting-mcp
last_verified: 2026/05/09
---

# Filler removal, British English, repetition collapse, basic formatting

> **Where you are:** [Architecture map](../README.md) → [LLM, Formatting, MCP](README.md) → Filler and British

**Plain English summary.** Four pure functions that work on a single string. Filler removal strips "um" / "uh" / "like" and friends. British English maps US spellings to UK (`organize` → `organise`, `color` → `colour`). Repetition collapse drops stutters and "I need I need to" doubles. `format_text` capitalises after sentence-ending punctuation and tidies spacing. All four are case-aware and word-boundary safe.

## At a glance

- Crate: `lumotia-ai-formatting`
- Path: `crates/ai-formatting/src/rule_based.rs` (also home to the anti-hallucination filter — that has its own page)
- LOC: 573 total in the file
- Public surface (relevant to this page):
  - `pub fn remove_fillers(text: &str) -> String` (`crates/ai-formatting/src/rule_based.rs:38`)
  - `pub fn collapse_repetitions(text: &str) -> String` (`crates/ai-formatting/src/rule_based.rs:69`) — not re-exported at crate root, called from `pipeline.rs`
  - `pub fn to_british_english(text: &str) -> String` (`crates/ai-formatting/src/rule_based.rs:190`)
  - `pub fn format_text(text: &str) -> String` (`crates/ai-formatting/src/rule_based.rs:238`)
- External deps that matter: `regex-lite = 0.1` (no lookbehinds, so we use explicit `\b` word boundaries).
- Tauri command that calls this (slice 2, best guess): not called directly. Reaches Tauri only via `post_process_segments` (`crates/ai-formatting/src/pipeline.rs:38`) — see [`formatting-pipeline.md`](formatting-pipeline.md).

## What's in here

### `remove_fillers` (`crates/ai-formatting/src/rule_based.rs:38`)

Twelve filler patterns compiled once into a `LazyLock<Vec<regex_lite::Regex>>` (`:6`):

```text
um, uh, er, ah, like, you know, sort of, kind of, I mean,
basically, actually, literally
```

Each is wrapped as `(?i)\b{escaped}\b[,.]?\s*`. Word-boundary on both sides, optional trailing comma or period, then any whitespace. The case-insensitive flag handles `Um`, `UH`, `Like`. The optional trailing punctuation handles "Like, this thing" → "this thing".

After substitution, runs of whitespace are collapsed in a single pass (no second regex pass — the loop at `:48` walks the string char by char). Final `trim()` removes leading or trailing whitespace introduced by removed leading fillers.

Tests `remove_fillers_strips_um_and_uh` (`:428`) and `remove_fillers_preserves_legitimate_words` (`:436`) cover the basic case and the "umbrella is not um" word-boundary case.

### `collapse_repetitions` (`crates/ai-formatting/src/rule_based.rs:69`)

Called from the pipeline only when `format_mode != Raw`. Collapses immediate repeated short phrases:

- `"I I can do that"` → `"I can do that"`
- `"I need I need to go"` → `"I need to go"`
- `"Think think that's that"` → `"Think that's that"`

Algorithm: tokenise on whitespace, normalise each token (`normalise_repetition_token` strips non-alphanumeric edges and lowercases), then a sliding-window over `kept_indices`. For window sizes 1, 2, 3 (longest first), check whether the just-kept window equals the upcoming window; if so, advance `i` past the upcoming window without keeping. Single-token doubles fall out of the same loop because the immediate-prev check at the bottom of the loop also handles `phrase_len == 1`.

The `1..=3` ceiling on phrase length is deliberate: longer "repeated phrases" usually are not stutters but legitimate emphasis, and the false-positive rate climbs.

### `to_british_english` (`crates/ai-formatting/src/rule_based.rs:190`)

Forty-something mappings in `BRITISH_REPLACEMENTS` (`:139`), grouped:

- `-ize` → `-ise` (and inflected forms): organize, recognize, realize, analyze, apologize, authorize, categorize, characterize, customize, digitize, emphasize, finalize, generalize, harmonize, initialize, maximize, minimize, modernize, normalize, optimize, prioritize, revolutionize, specialize, standardize, summarize, utilize.
- `-or` → `-our`: color, favor, honor, humor, labor, neighbor, behavior.
- `-er` → `-re`: center, fiber, liter, meter, theater.
- `-ense` → `-ence`: defense, offense.
- Other: catalog → catalogue, dialog → dialogue.

Each entry is a plain ASCII base word. `to_british_english` wraps it as `(?i)\b{escaped}(?:d|s|r|rs)?\b` so inflected forms (`organized`, `organizes`, `organizer`, `organizers`) match without separate entries. The replacement closure preserves capitalisation: if the matched first character is uppercase, the British replacement's first character is uppercased too.

A `debug_assert!` enforces ASCII-only entries and matched text — byte-indexing the suffix is only safe under that assumption.

Tests cover the inflected case (`to_british_english_converts_ize_to_ise` at `:444`), capitalisation preservation (`to_british_english_preserves_case` at `:450`), and the `-our` family (`to_british_english_handles_colour` at `:457`).

### `format_text` (`crates/ai-formatting/src/rule_based.rs:238`)

Lightweight pass: capitalise after `.`, `!`, `?`, `\n`. Collapse double spaces. The first letter of the input is also capitalised.

Implementation walks the input as `Vec<char>` to handle multi-byte safely. The double-space collapse is a peek-ahead on `chars[i + 1]`. Empty input returns empty.

## Data flow

```
input: &str
remove_fillers:
  → for each regex: replace_all → " "
  → manual whitespace-run collapse
  → trim
  → return String

collapse_repetitions:
  → tokenise on whitespace
  → normalise each token (lowercase, strip edges)
  → sliding-window dedupe (lengths 3, 2, 1)
  → join kept tokens with " "
  → trim
  → return String

to_british_english:
  → for each (us, uk):
       compile regex r"(?i)\b{us}(?:d|s|r|rs)?\b"
       replace_all with closure that preserves capitalisation
  → return String

format_text:
  → walk chars, collapse double spaces, capitalise after . ! ? \n
  → return String
```

All four are pure: they take `&str`, return `String`, no mutation, no IO, no lock contention.

## Watch-outs

- **Filler word boundaries depend on `regex-lite`'s `\b` semantics.** `regex-lite` has no lookbehind / lookahead support, so "kind of" is matched as a contiguous phrase; we cannot do "kind of" only when followed by a noun. False positives on "kind of bread" (which becomes "bread") are accepted as the cost of catching every "kind of" filler. Worth knowing if reports of stripped-meaning come in.
- **Repetition collapse is whitespace-tokenised.** Punctuation attached to a token ("hello,") becomes part of the token before normalisation. The normaliser strips edges before comparison so "hello," and "hello" compare equal. The output preserves the original token (with punctuation), so "Hello hello, world" collapses to "Hello world" — not "Hello, world".
- **British English regex compiles per call.** Forty-ish entries each compile a fresh `regex_lite::Regex` per `to_british_english` call. Not free, but `regex-lite` is small and the call is per-segment, not per-token. If profiling ever flags this, hoist into a `LazyLock<Vec<(Regex, &str)>>`.
- **Capitalisation preservation only handles ASCII first chars.** All entries are ASCII; the `debug_assert!` at `:209` enforces it. A future entry like "naïve" would need to revisit the byte-slice logic.
- **`format_text` does not preserve double newlines.** The peek-ahead at `:252` collapses any double-space, but `\n` characters pass through verbatim. The pipeline's smart-paragraph step prepends `\n\n` *after* `format_text` runs (see [`formatting-pipeline.md`](formatting-pipeline.md) step ordering), so paragraph breaks are not lost.
- **Inflected suffix list is `(d|s|r|rs)?`, not exhaustive.** Past tense `-ed` works, plural `-s` works, agent `-r` and plural agent `-rs` work. `-ing` is not in the list because the British base words ending in `-ise` already form `-ising` regularly, and the `-ize` → `-ise` substitution handles "organizing" via the closure (the matched suffix would be `ing`, which the closure preserves verbatim) — except the suffix regex does not include `ing`, so `organizing` does not match. **This is a known gap**: `-ing` forms of `-ize` verbs slip through unconverted. Either add `|ing` to the suffix alternation or accept it. Worth flagging.

## See also

- [Pipeline overview](formatting-pipeline.md)
- [Anti-hallucination filter (also in rule_based.rs)](formatting-anti-hallucination.md)
- [Slice README](README.md)