Lumotia/docs/architecture-map/04-llm-formatting-mcp/formatting-correction-learning.md

name, type, slice, last_verified

name	type	slice	last_verified
Correction learning (HITL → custom dictionary)	architecture-map-page	04-llm-formatting-mcp	2026/05/09

Correction learning (HITL → custom dictionary)

Where you are: Architecture map → LLM, Formatting, MCP → Correction learning

Plain English summary. When a user edits a transcript, extract_corrections infers which words were swapped and surfaces them as candidates for the user's custom-vocabulary dictionary. Conservative — only short, low-edit-distance, non-existing-term substitutions count. Large rewrites are ignored. The result feeds back into the LLM cleanup prompt's dictionary suffix.

At a glance

• Crate: lumotia-ai-formatting
• Path: crates/ai-formatting/src/correction_learning.rs
• LOC: 229
• Public surface:
  • pub fn extract_corrections(original_text: &str, edited_text: &str, existing_terms: &[String]) -> Vec<String> (crates/ai-formatting/src/correction_learning.rs:131)
  • Re-exported at crate root as lumotia_ai_formatting::extract_corrections (crates/ai-formatting/src/lib.rs:7).
• External deps that matter: none — pure CPU, pure stdlib.
• Tauri command that calls this (slice 2, best guess): commands::profiles.rs:14 imports it; the actual call is at src-tauri/src/commands/profiles.rs:165. The wrapper command is named in profiles.rs (likely learn_corrections_*_cmd — check slice 2's profiles page).

What's in here

Constants (crates/ai-formatting/src/correction_learning.rs:3)

• MAX_REWRITE_RATIO = 0.5 — if more than half the original words got substituted, treat as a rewrite, not corrections.
• MIN_CORRECTION_LEN = 3 — corrected words shorter than 3 chars are dropped (avoids noise on "a", "of", "to").
• MAX_DISTANCE_RATIO = 0.65 — Levenshtein distance over max-length must be at most 0.65; anything more is a different word, not a correction.
• MAX_CORRECTIONS_PER_EDIT = 8 — cap on how many correction candidates a single edit can yield.

Levenshtein distance (crates/ai-formatting/src/correction_learning.rs:8)

Standard two-row dynamic programming distance over Vec<char>. Used to decide whether Shunade → Sinead is "phonetic correction" (low distance ratio) or "different word" (high).

tokenize and trim_non_word_edges (:29, :33)

Whitespace split, then strip non-alphanumeric edges (keeping _ for code). "hello," → "hello". Empty results filtered out.

find_edited_region (:42)

If the edit is significantly larger than the original (field_value.len() > original.len() * 1.5), the user might have appended notes around the original. The function tries to locate the region in the edited text that aligns with the original via a sliding-window word-match, returning just that region. If the alignment score is below 30% of the window size, the function gives up and returns the full edited text. The original-text-contained-as-substring case short-circuits: if the original appears verbatim, no corrections were made — return the original.

This guards against the failure mode where a user dictates a paragraph, then later appends 500 words of new content; we do not want to "learn" every word of the appended content as a correction.

find_substitutions (:81)

LCS-based alignment between original and edited word lists. Walks back through the DP table to produce an aligned Vec<(Option<String>, Option<String>)>. A pair (Some(orig), None) is a deletion; (None, Some(edit)) is an insertion. A (deletion, insertion) adjacent pair is a substitution — the only shape this function emits as a (orig, corrected) tuple.

The substitution-detection condition (:118) is precise: (Some(orig_word), None, None, Some(corrected_word)) across two consecutive aligned pairs. Adjacent inserts and adjacent deletes do not become substitutions.

extract_corrections (:131)

The pipeline:

1. Empty / unchanged guard. Return empty when either side is whitespace-only or when texts are identical.
2. Locate edited region via find_edited_region. If the region equals the original, return empty (nothing was edited in this region).
3. Tokenise both sides. Empty token lists → return empty.
4. Find substitutions via the LCS aligner.
5. Rewrite-ratio gate. If substitutions count exceeds MAX_REWRITE_RATIO of original word count, return empty. Catches the "user threw away the transcript and rewrote it" case.
6. Build the existing-terms set (lowercased) for the dedupe step.
7. For each substitution:
   • Skip if orig.lower() == corrected.lower() (same word, just casing).
   • Skip if corrected.len() < MIN_CORRECTION_LEN.
   • Skip if corrected.lower() is already in existing_terms.
   • Skip if we already produced this correction in this batch.
   • Compute edit distance, skip if distance / max(orig.len, corrected.len) > MAX_DISTANCE_RATIO.
   • Push the corrected word (preserving its casing) onto the result.
   • Stop at MAX_CORRECTIONS_PER_EDIT.

Returns Vec<String> of new terms the caller should consider adding to the user's custom-vocabulary dictionary.

Tests (crates/ai-formatting/src/correction_learning.rs:193)

• extracts_phonetic_corrections_for_profile_learning (:197) — "Shunade" → "Sinead" is detected.
• ignores_large_rewrites (:208) — total rewrite returns empty.
• skips_terms_already_in_profile_dictionary (:219) — "Corble" → "CORBEL" is dropped because CORBEL is already in existing_terms.

Data flow

(original_text, edited_text, existing_terms)
  → empty / unchanged guard
  → edited_region = find_edited_region(original_text, edited_text)
       (handles "user appended new content" case)
  → tokenise both
  → substitutions = find_substitutions(original_words, edited_words)  (LCS-based)
  → rewrite-ratio gate (drop if > 50% substituted)
  → for each substitution:
       skip same-casing, skip too-short, skip existing, skip duplicate-this-batch,
       skip if distance/max_len > 0.65
  → cap at 8 results
  → return Vec<String> of corrected words (case preserved)

caller (Tauri profiles command) merges these into the user's profile.dictionary,
which then appears as PostProcessOptions.dictionary_terms in the next pipeline run.

Watch-outs

• Conservative by design. Many false positives is worse than a few false negatives, because every "learned" word goes into the cleanup prompt and biases the LLM. The four constants combine to keep the noise floor low. Tuning them down (looser distance ratio, smaller min length) is safe to experiment with but should be A/B-tested on real edits.
• Levenshtein on words. Distance is computed character-by-character on the lowercased forms, not phoneme-by-phoneme. Shunade → Sinead (distance 4, max-len 7, ratio 0.57) makes it under the 0.65 cap. Run → Walk (distance 4, max-len 4, ratio 1.0) does not.
• Casing is preserved in the output. The dedupe set is lowercased (so two corrections that differ only in casing collapse to one), but the kept variant is whatever case the user typed. If a user types Sinead and SINEAD in the same edit, the first one wins.
• existing_terms should pass the user's full custom dictionary. The function lowercases internally; the caller can pass any casing. Filling this in correctly is the difference between "we keep suggesting CORBEL the user already added" and "we only suggest new terms".
• Substitutions only — no insertions or deletions. A user inserting a new word that the ASR did not pick up at all is not a "correction" to anything; it is content. The aligner correctly leaves it as (None, Some(_)) and the subsequent matcher ignores those.
• No language awareness. The function does not know about morphological variants. "organize" → "organise" is correctly detected as a 1-character substitution (because to_british_english would have done it earlier in the pipeline), but if a user edits a transcript Lumotia did not Britanise, the function will happily learn organise as a new term — duplicating work.
• find_edited_region heuristic floor is 30%. Below that match-score, the function gives up alignment and returns the whole edited text, which is then very likely to fail the rewrite-ratio gate. Effectively a graceful "I don't know what changed, skip it" path.

See also

• LLM cleanup bridge — where dictionary terms get used
• Pipeline overview — where dictionary terms enter the pipeline
• Slice README