Lumotia/docs/architecture-map/04-llm-formatting-mcp/formatting-filler-and-british.md

name, type, slice, last_verified

name	type	slice	last_verified
Filler removal, British English, repetition collapse, basic formatting	architecture-map-page	04-llm-formatting-mcp	2026/05/09

Filler removal, British English, repetition collapse, basic formatting

Where you are: Architecture map → LLM, Formatting, MCP → Filler and British

Plain English summary. Four pure functions that work on a single string. Filler removal strips "um" / "uh" / "like" and friends. British English maps US spellings to UK (organize → organise, color → colour). Repetition collapse drops stutters and "I need I need to" doubles. format_text capitalises after sentence-ending punctuation and tidies spacing. All four are case-aware and word-boundary safe.

At a glance

• Crate: lumotia-ai-formatting
• Path: crates/ai-formatting/src/rule_based.rs (also home to the anti-hallucination filter — that has its own page)
• LOC: 573 total in the file
• Public surface (relevant to this page):
  • pub fn remove_fillers(text: &str) -> String (crates/ai-formatting/src/rule_based.rs:38)
  • pub fn collapse_repetitions(text: &str) -> String (crates/ai-formatting/src/rule_based.rs:69) — not re-exported at crate root, called from pipeline.rs
  • pub fn to_british_english(text: &str) -> String (crates/ai-formatting/src/rule_based.rs:190)
  • pub fn format_text(text: &str) -> String (crates/ai-formatting/src/rule_based.rs:238)
• External deps that matter: regex-lite = 0.1 (no lookbehinds, so we use explicit \b word boundaries).
• Tauri command that calls this (slice 2, best guess): not called directly. Reaches Tauri only via post_process_segments (crates/ai-formatting/src/pipeline.rs:38) — see formatting-pipeline.md.

What's in here

remove_fillers (crates/ai-formatting/src/rule_based.rs:38)

Twelve filler patterns compiled once into a LazyLock<Vec<regex_lite::Regex>> (:6):

um, uh, er, ah, like, you know, sort of, kind of, I mean,
basically, actually, literally

Each is wrapped as (?i)\b{escaped}\b[,.]?\s*. Word-boundary on both sides, optional trailing comma or period, then any whitespace. The case-insensitive flag handles Um, UH, Like. The optional trailing punctuation handles "Like, this thing" → "this thing".

After substitution, runs of whitespace are collapsed in a single pass (no second regex pass — the loop at :48 walks the string char by char). Final trim() removes leading or trailing whitespace introduced by removed leading fillers.

Tests remove_fillers_strips_um_and_uh (:428) and remove_fillers_preserves_legitimate_words (:436) cover the basic case and the "umbrella is not um" word-boundary case.

collapse_repetitions (crates/ai-formatting/src/rule_based.rs:69)

Called from the pipeline only when format_mode != Raw. Collapses immediate repeated short phrases:

• "I I can do that" → "I can do that"
• "I need I need to go" → "I need to go"
• "Think think that's that" → "Think that's that"

Algorithm: tokenise on whitespace, normalise each token (normalise_repetition_token strips non-alphanumeric edges and lowercases), then a sliding-window over kept_indices. For window sizes 1, 2, 3 (longest first), check whether the just-kept window equals the upcoming window; if so, advance i past the upcoming window without keeping. Single-token doubles fall out of the same loop because the immediate-prev check at the bottom of the loop also handles phrase_len == 1.

The 1..=3 ceiling on phrase length is deliberate: longer "repeated phrases" usually are not stutters but legitimate emphasis, and the false-positive rate climbs.

to_british_english (crates/ai-formatting/src/rule_based.rs:190)

Forty-something mappings in BRITISH_REPLACEMENTS (:139), grouped:

• -ize → -ise (and inflected forms): organize, recognize, realize, analyze, apologize, authorize, categorize, characterize, customize, digitize, emphasize, finalize, generalize, harmonize, initialize, maximize, minimize, modernize, normalize, optimize, prioritize, revolutionize, specialize, standardize, summarize, utilize.
• -or → -our: color, favor, honor, humor, labor, neighbor, behavior.
• -er → -re: center, fiber, liter, meter, theater.
• -ense → -ence: defense, offense.
• Other: catalog → catalogue, dialog → dialogue.

Each entry is a plain ASCII base word. to_british_english wraps it as (?i)\b{escaped}(?:d|s|r|rs)?\b so inflected forms (organized, organizes, organizer, organizers) match without separate entries. The replacement closure preserves capitalisation: if the matched first character is uppercase, the British replacement's first character is uppercased too.

A debug_assert! enforces ASCII-only entries and matched text — byte-indexing the suffix is only safe under that assumption.

Tests cover the inflected case (to_british_english_converts_ize_to_ise at :444), capitalisation preservation (to_british_english_preserves_case at :450), and the -our family (to_british_english_handles_colour at :457).

format_text (crates/ai-formatting/src/rule_based.rs:238)

Lightweight pass: capitalise after ., !, ?, \n. Collapse double spaces. The first letter of the input is also capitalised.

Implementation walks the input as Vec<char> to handle multi-byte safely. The double-space collapse is a peek-ahead on chars[i + 1]. Empty input returns empty.

Data flow

input: &str
remove_fillers:
  → for each regex: replace_all → " "
  → manual whitespace-run collapse
  → trim
  → return String

collapse_repetitions:
  → tokenise on whitespace
  → normalise each token (lowercase, strip edges)
  → sliding-window dedupe (lengths 3, 2, 1)
  → join kept tokens with " "
  → trim
  → return String

to_british_english:
  → for each (us, uk):
       compile regex r"(?i)\b{us}(?:d|s|r|rs)?\b"
       replace_all with closure that preserves capitalisation
  → return String

format_text:
  → walk chars, collapse double spaces, capitalise after . ! ? \n
  → return String

All four are pure: they take &str, return String, no mutation, no IO, no lock contention.

Watch-outs

• Filler word boundaries depend on regex-lite's \b semantics. regex-lite has no lookbehind / lookahead support, so "kind of" is matched as a contiguous phrase; we cannot do "kind of" only when followed by a noun. False positives on "kind of bread" (which becomes "bread") are accepted as the cost of catching every "kind of" filler. Worth knowing if reports of stripped-meaning come in.
• Repetition collapse is whitespace-tokenised. Punctuation attached to a token ("hello,") becomes part of the token before normalisation. The normaliser strips edges before comparison so "hello," and "hello" compare equal. The output preserves the original token (with punctuation), so "Hello hello, world" collapses to "Hello world" — not "Hello, world".
• British English regex compiles per call. Forty-ish entries each compile a fresh regex_lite::Regex per to_british_english call. Not free, but regex-lite is small and the call is per-segment, not per-token. If profiling ever flags this, hoist into a LazyLock<Vec<(Regex, &str)>>.
• Capitalisation preservation only handles ASCII first chars. All entries are ASCII; the debug_assert! at :209 enforces it. A future entry like "naïve" would need to revisit the byte-slice logic.
• format_text does not preserve double newlines. The peek-ahead at :252 collapses any double-space, but \n characters pass through verbatim. The pipeline's smart-paragraph step prepends \n\n after format_text runs (see formatting-pipeline.md step ordering), so paragraph breaks are not lost.
• Inflected suffix list is (d|s|r|rs)?, not exhaustive. Past tense -ed works, plural -s works, agent -r and plural agent -rs work. -ing is not in the list because the British base words ending in -ise already form -ising regularly, and the -ize → -ise substitution handles "organizing" via the closure (the matched suffix would be ing, which the closure preserves verbatim) — except the suffix regex does not include ing, so organizing does not match. This is a known gap: -ing forms of -ize verbs slip through unconverted. Either add |ing to the suffix alternation or accept it. Worth flagging.

See also

• Pipeline overview
• Anti-hallucination filter (also in rule_based.rs)
• Slice README