Jake
53fe848979
New crates/ai-formatting/src/to_plain_text.rs module with one public function: to_plain_text(&[Segment]) -> String. Rules the function enforces: - each segment's text is whitespace-normalised (any run of unicode whitespace collapses to a single ASCII space, so tabs, newlines, and NBSPs never reach the LLM), - empty and whitespace-only segments are dropped, - remaining segments are joined with a single ASCII space, - the joined string is normalised again (so a segment ending in a space followed by one starting in a space does not produce a double space) and trimmed end-to-end. pipeline.rs's inline join is replaced with this call. Whisper's timestamp fields (Segment.start / .end) are carried separately and never reach the LLM by construction — the "timestamps stripped" half of brief item #29's acceptance falls out of using Segment.text alone. The work the module actually adds is whitespace discipline and the tested boundary (empty input, empty-only input, NBSPs, pathological whitespace runs, idempotence, double-space at join boundaries). Source: Scriberr PR #288 — feeding raw Whisper JSON (with timestamps and per-segment structure) degraded cleanup quality; plain-text input raised it back.
12 lines
432 B
Rust
12 lines
432 B
Rust
pub mod correction_learning;
|
|
mod llm_client;
|
|
pub mod pipeline;
|
|
pub mod rule_based;
|
|
pub mod to_plain_text;
|
|
|
|
pub use correction_learning::extract_corrections;
|
|
pub use llm_client::{cleanup_text as llm_cleanup_text, LlmPromptPreset};
|
|
pub use pipeline::{post_process_segments, FormatMode, PostProcessOptions};
|
|
pub use rule_based::{format_text, is_hallucination, remove_fillers, to_british_english};
|
|
pub use to_plain_text::to_plain_text;
|