feat(A.4 #29): plain-text pre-formatter before LLM cleanup
New crates/ai-formatting/src/to_plain_text.rs module with one public function: to_plain_text(&[Segment]) -> String. Rules the function enforces: - each segment's text is whitespace-normalised (any run of unicode whitespace collapses to a single ASCII space, so tabs, newlines, and NBSPs never reach the LLM), - empty and whitespace-only segments are dropped, - remaining segments are joined with a single ASCII space, - the joined string is normalised again (so a segment ending in a space followed by one starting in a space does not produce a double space) and trimmed end-to-end. pipeline.rs's inline join is replaced with this call. Whisper's timestamp fields (Segment.start / .end) are carried separately and never reach the LLM by construction — the "timestamps stripped" half of brief item #29's acceptance falls out of using Segment.text alone. The work the module actually adds is whitespace discipline and the tested boundary (empty input, empty-only input, NBSPs, pathological whitespace runs, idempotence, double-space at join boundaries). Source: Scriberr PR #288 — feeding raw Whisper JSON (with timestamps and per-segment structure) degraded cleanup quality; plain-text input raised it back.
This commit is contained in:
@@ -2,8 +2,10 @@ pub mod correction_learning;
|
||||
mod llm_client;
|
||||
pub mod pipeline;
|
||||
pub mod rule_based;
|
||||
pub mod to_plain_text;
|
||||
|
||||
pub use correction_learning::extract_corrections;
|
||||
pub use llm_client::{cleanup_text as llm_cleanup_text, LlmPromptPreset};
|
||||
pub use pipeline::{post_process_segments, FormatMode, PostProcessOptions};
|
||||
pub use rule_based::{format_text, is_hallucination, remove_fillers, to_british_english};
|
||||
pub use to_plain_text::to_plain_text;
|
||||
|
||||
Reference in New Issue
Block a user