Review feedback (MINOR): char::is_whitespace returns false for
zero-width format codepoints (U+200B ZWSP, U+200C ZWNJ, U+200D ZWJ,
U+2060 WORD JOINER, U+FEFF ZWNBSP / BOM). The original normalise
pass let them through to the LLM where they waste tokens without
contributing any natural-language content.
Makes the decision explicit: these chars STRIP entirely rather than
collapse to a space. Collapsing would silently insert a word break
where the source had none ("hello<FEFF>world" → "hello world"
would merge two words into a space-separated pair that the original
author did not intend). Stripping preserves the original token
boundaries and drops the invisible noise.
Three new tests:
- zero_width_format_chars_strip_entirely — exhaustive coverage of
all five handled codepoints.
- zero_width_chars_do_not_break_adjacent_whitespace_collapsing —
"hello <FEFF> world" still collapses to "hello world" (the
strip does not leave behind an artefact that breaks the whitespace
collapse pass).
- leading_bom_is_stripped — a BOM at segment start, the common
artefact pattern when Whisper consumes an encoded file.
Jake
e54f0404ce