Lumotia/docs/architecture-map/03-audio-transcription/transcription-streaming.md

name, type, slice, last_verified

name	type	slice	last_verified
Streaming primitives (VAD chunker, LocalAgreement, buffer trim)	architecture-map-page	03-audio-transcription	2026/05/09

Streaming primitives

Where you are: Architecture map → Audio + Transcription → Streaming primitives

Plain English summary. Live capture cannot just feed Whisper a single 30-minute buffer at session end. Three primitives cooperate to produce responsive live transcription: RmsVadChunker decides when "speech" has ended and emits a VadChunk; LocalAgreement holds tokens as tentative until two consecutive Whisper passes agree on a prefix, then commits that prefix; trim_buffer_to_commit_point drains committed audio out of the working buffer so memory stays bounded. The trait surface (VadChunker) matches what a future Silero implementation will provide, so when the ort version conflict resolves, the live session keeps working.

At a glance

• Crate: lumotia-transcription
• Paths:
  • crates/transcription/src/streaming/mod.rs (84 LOC) — trait + types + re-exports.
  • crates/transcription/src/streaming/rms_vad.rs (736 LOC) — RmsVadChunker.
  • crates/transcription/src/streaming/commit_policy.rs (404 LOC) — LocalAgreement, Token, CommitDecision, CommitPolicy.
  • crates/transcription/src/streaming/buffer_trim.rs (208 LOC) — sample_index_for_seconds, trim_buffer_to_commit_point.
• External deps: none beyond stdlib.
• Internal callers (best effort, slice 2 reconciles): scheduled to be wired into src-tauri/src/commands/live.rs (brief items #21, #24, #25). Comments note the integration ships in follow-up commits so threshold tuning can be validated against real microphone captures.

Public surface:

• pub trait VadChunker: Send — crates/transcription/src/streaming/mod.rs:44. Methods: push, flush, reset, next_sample_index.
• pub struct VadChunk { start_sample: u64, samples: Vec<f32> } — crates/transcription/src/streaming/mod.rs:22.
• pub struct RmsVadChunker — crates/transcription/src/streaming/rms_vad.rs:57.
• pub fn RmsVadChunker::new() -> Self — crates/transcription/src/streaming/rms_vad.rs:90.
• pub fn RmsVadChunker::with_thresholds(...) -> Self — crates/transcription/src/streaming/rms_vad.rs:100.
• pub fn RmsVadChunker::sample_rate_hz(&self) -> u32 — crates/transcription/src/streaming/rms_vad.rs:128.
• pub struct Token { text, start_secs, end_secs } — crates/transcription/src/streaming/commit_policy.rs:28. PartialEq is text-only.
• pub enum CommitPolicy { LocalAgreement { n: usize } } — crates/transcription/src/streaming/commit_policy.rs:57. Default is n = 2.
• pub struct CommitDecision { newly_committed: Vec<Token>, tentative: Vec<Token> } — crates/transcription/src/streaming/commit_policy.rs:44.
• pub struct LocalAgreement — crates/transcription/src/streaming/commit_policy.rs:77.
• pub fn LocalAgreement::{new, from_policy, push, flush, last_committed_end_secs, reset} — same file.
• pub fn sample_index_for_seconds(end_secs: f64, sample_rate: u32) -> u64 — crates/transcription/src/streaming/buffer_trim.rs:23.
• pub fn trim_buffer_to_commit_point(buffer: &mut Vec<f32>, buffer_start_sample: u64, commit_sample_index: u64) -> u64 — crates/transcription/src/streaming/buffer_trim.rs:40.

What's in here

VadChunker trait (streaming/mod.rs:44)

Send-bound so a chunker can be moved into spawn_blocking. Object-safe (compile-time witness at streaming/mod.rs:77). Three methods plus an absolute-position accessor:

• push(&mut self, samples: &[f32]) -> Vec<VadChunk> — feed samples, get any chunks ready to dispatch.
• flush(&mut self) -> Vec<VadChunk> — end-of-session, surface any in-flight speech (note: returns Vec, not Option, because the zero-padded final frame can legitimately trigger both a mid-flush emission and a closing emission).
• reset(&mut self) — drop accumulated state.
• next_sample_index(&self) -> u64 — what the next sample fed via push will be at, used by the commit-policy / buffer-trim glue.

RmsVadChunker (streaming/rms_vad.rs)

Energy-backed VAD with hysteresis. Constants:

• FRAME_SAMPLES = 800 (50 ms at 16 kHz).
• DEFAULT_ENTER_RMS_THRESHOLD = 0.003, DEFAULT_EXIT_RMS_THRESHOLD = 0.0014 — separate thresholds give hysteresis so a score bouncing on a single value doesn't toggle state every frame.
• DEFAULT_SPEECH_ONSET_FRAMES = 3 (3 sustained frames over enter threshold required to enter speech state — filters keyboard clicks).
• DEFAULT_SILENCE_CLOSE_SAMPLES = 8_000 (500 ms of sub-threshold to close).
• DEFAULT_MAX_CHUNK_SAMPLES = 32_000 (2 s, hard cap so Whisper isn't fed a 30-second monolith).
• DEFAULT_SAMPLE_RATE_HZ = 16_000.

Internal state machine (streaming/rms_vad.rs:46): Idle ↔ InSpeech. The onset_buffer keeps a rolling speech_onset_frames * FRAME_SAMPLES worth of audio so when speech is confirmed, the emitted chunk includes the onset itself, not just the post-onset audio.

Two emit paths:

• emit_active_chunk_and_close (streaming/rms_vad.rs:217) — end-of-utterance. Trims trailing silence, returns to Idle.
• emit_active_chunk_continue (streaming/rms_vad.rs:248) — hit max_chunk_samples mid-speech. Stays in InSpeech, advances active_chunk_start by the emitted length so the next chunk's start sample is contiguous.

flush is unusually careful (streaming/rms_vad.rs:294): it pads any sub-frame pending buffer to a full frame with zeros, runs consume_frame on it (which may emit a chunk), then if state is still InSpeech emits the active chunk as a closing chunk. Both emissions are returned. A 2026-04-22 audit (CRITICAL C2) found the prior code dropped the consume_frame chunk via let _ = consume_frame(...); regression tests at streaming/rms_vad.rs:567 and :614 keep this honest.

LocalAgreement (streaming/commit_policy.rs)

Source: ufal/whisper_streaming. Algorithm:

• Hold the last n ASR passes (default n = 2).
• The longest common prefix of those passes is the "agreed" prefix.
• Any tokens beyond that prefix are tentative (UI dashed-underline per workstream-B).
• Commit only grows: once a token is committed, even a divergent later pass cannot uncommit it (new_committed = lcp_len.max(self.committed_count)).

Token equality is text-only (streaming/commit_policy.rs:34); timestamps drift slightly between overlapping Whisper windows. Start/end seconds are absolute (session-relative) so the buffer-trim layer can compute sample indices.

CommitDecision has two vecs:

• newly_committed — append to the frontend's committed list.
• tentative — replaces (not appends to) the previous tentative list.

flush (streaming/commit_policy.rs:163) is end-of-stream: anything still tentative in the latest pass is committed and returned. Updates last_committed_end_secs so the buffer trim works on the closing region.

Defensive bookkeeping at streaming/commit_policy.rs:131: a later pass can legitimately arrive shorter than committed_count (Whisper re-transcribing an overlapping window with fewer segments). All slice indices are clamped against latest.len() to avoid panics. Regression test at streaming/commit_policy.rs:317.

Buffer trim (streaming/buffer_trim.rs)

sample_index_for_seconds(end_secs, sample_rate) -> u64:

• Returns 0 for non-finite or <= 0 inputs (defensive — without is_finite, f64::INFINITY would saturate to u64::MAX and trim the whole buffer forever).
• (end_secs * sample_rate as f64).round() as u64 otherwise.

trim_buffer_to_commit_point(buffer, buffer_start_sample, commit_sample_index) -> u64:

• If the commit point is at or before the current buffer origin, do nothing.
• If it's past the buffer end, clear the buffer and park the new origin at the commit point.
• Otherwise drain the prefix and return the new origin.

The integrated property test at streaming/buffer_trim.rs:144 simulates 100 cycles of 16 kHz captures and proves the buffer envelope stays bounded by 2 * tentative_per_cycle instead of growing linearly.

Data flow

StreamingResampler 16 kHz mono
   └─ capture_buffer (Vec<f32>, anchored at buffer_start_sample)
       └─ RmsVadChunker.push(samples) → Vec<VadChunk>
           └─ for each VadChunk:
                 spawn_blocking → LocalEngine.transcribe_sync(chunk)
                     → Vec<Segment>
                         → tokens (Token { text, start_secs, end_secs })
                             → LocalAgreement.push(tokens) → CommitDecision
                                 ├─ newly_committed → emit to frontend
                                 │   └─ LocalAgreement.last_committed_end_secs
                                 │       → sample_index_for_seconds
                                 │           → trim_buffer_to_commit_point(capture_buffer)
                                 └─ tentative → replace tentative tail in frontend
…session ends…
   └─ RmsVadChunker.flush() → final Vec<VadChunk>
   └─ LocalAgreement.flush() → final tentative committed

Watch-outs

• Not yet wired into live.rs. All three primitives are unit-tested but their integration into src-tauri/src/commands/live.rs is explicitly described as a follow-up. Slice 2 owns the verification of whether that has landed.
• Token equality is text-only. Two passes that produce the same word with different casing or punctuation will not match. Whisper is fairly stable here, but a model swap is worth watching.
• speech_onset_frames is a transient filter, not a denoiser. Sustained background talk above the enter threshold will register as speech.
• FRAME_SAMPLES = 800 assumes 16 kHz. A future backend at 24 kHz would need a different frame size to hit the same 50 ms window. RmsVadChunker::sample_rate_hz returns the constant; do not lie to it.
• reset() zeros next_sample_index but flush() deliberately does not (so a flush at end-of-session leaves the running counter accurate for any post-flush diagnostic).
• max_chunk_samples is a hard 2 s cap. A user reading aloud without a breath crosses it constantly. The continue-emit logic preserves audio contiguity (regression test at streaming/rms_vad.rs:476), but the resulting chunk boundaries land mid-word, which the LocalAgreement two-pass overlap can stitch back together — if the live session is feeding overlapping windows. If live.rs ever feeds non-overlapping chunks, expect words sliced in half.
• LocalAgreement::flush may double-commit. Calling it twice on the same instance: the second call would see latest.len() == committed_count and return empty. The flush_with_no_history_is_empty test pins this. Callers should still only call once at session end.
• Buffer-trim's sample_index_for_seconds rounds nearest. Integer comparisons against the trimmed buffer must not assume floor.

See also

• Audio capture pipeline — produces the chunks fed into StreamingResampler.
• Audio resampling — StreamingResampler upstream of the chunker.
• Transcription engines overview — LocalEngine.transcribe_sync is what the streaming layer calls per chunk.
• Audio VAD — the deferred Silero path that will eventually replace RmsVadChunker.
• docs/whisper-ecosystem/workstream-A.md, workstream-B.md — the design source for thresholds and commit/tentative UX.
• Tests in streaming/rms_vad.rs:359, streaming/commit_policy.rs:212, streaming/buffer_trim.rs:57.