Lumotia/docs/architecture-map/03-audio-transcription/transcription-streaming.md

---
name: Streaming primitives (VAD chunker, LocalAgreement, buffer trim)
type: architecture-map-page
slice: 03-audio-transcription
last_verified: 2026/05/09
---

# Streaming primitives

> **Where you are:** [Architecture map](../README.md) → [Audio + Transcription](README.md) → Streaming primitives

**Plain English summary.** Live capture cannot just feed Whisper a single 30-minute buffer at session end. Three primitives cooperate to produce responsive live transcription: `RmsVadChunker` decides when "speech" has ended and emits a `VadChunk`; `LocalAgreement` holds tokens as tentative until two consecutive Whisper passes agree on a prefix, then commits that prefix; `trim_buffer_to_commit_point` drains committed audio out of the working buffer so memory stays bounded. The trait surface (`VadChunker`) matches what a future Silero implementation will provide, so when the `ort` version conflict resolves, the live session keeps working.

## At a glance

- Crate: `lumotia-transcription`
- Paths:
  - `crates/transcription/src/streaming/mod.rs` (84 LOC) — trait + types + re-exports.
  - `crates/transcription/src/streaming/rms_vad.rs` (736 LOC) — `RmsVadChunker`.
  - `crates/transcription/src/streaming/commit_policy.rs` (404 LOC) — `LocalAgreement`, `Token`, `CommitDecision`, `CommitPolicy`.
  - `crates/transcription/src/streaming/buffer_trim.rs` (208 LOC) — `sample_index_for_seconds`, `trim_buffer_to_commit_point`.
- External deps: none beyond stdlib.
- Internal callers (best effort, slice 2 reconciles): scheduled to be wired into `src-tauri/src/commands/live.rs` (brief items #21, #24, #25). Comments note the integration ships in follow-up commits so threshold tuning can be validated against real microphone captures.

Public surface:

- `pub trait VadChunker: Send` — `crates/transcription/src/streaming/mod.rs:44`. Methods: `push`, `flush`, `reset`, `next_sample_index`.
- `pub struct VadChunk { start_sample: u64, samples: Vec<f32> }` — `crates/transcription/src/streaming/mod.rs:22`.
- `pub struct RmsVadChunker` — `crates/transcription/src/streaming/rms_vad.rs:57`.
- `pub fn RmsVadChunker::new() -> Self` — `crates/transcription/src/streaming/rms_vad.rs:90`.
- `pub fn RmsVadChunker::with_thresholds(...) -> Self` — `crates/transcription/src/streaming/rms_vad.rs:100`.
- `pub fn RmsVadChunker::sample_rate_hz(&self) -> u32` — `crates/transcription/src/streaming/rms_vad.rs:128`.
- `pub struct Token { text, start_secs, end_secs }` — `crates/transcription/src/streaming/commit_policy.rs:28`. PartialEq is text-only.
- `pub enum CommitPolicy { LocalAgreement { n: usize } }` — `crates/transcription/src/streaming/commit_policy.rs:57`. Default is `n = 2`.
- `pub struct CommitDecision { newly_committed: Vec<Token>, tentative: Vec<Token> }` — `crates/transcription/src/streaming/commit_policy.rs:44`.
- `pub struct LocalAgreement` — `crates/transcription/src/streaming/commit_policy.rs:77`.
- `pub fn LocalAgreement::{new, from_policy, push, flush, last_committed_end_secs, reset}` — same file.
- `pub fn sample_index_for_seconds(end_secs: f64, sample_rate: u32) -> u64` — `crates/transcription/src/streaming/buffer_trim.rs:23`.
- `pub fn trim_buffer_to_commit_point(buffer: &mut Vec<f32>, buffer_start_sample: u64, commit_sample_index: u64) -> u64` — `crates/transcription/src/streaming/buffer_trim.rs:40`.

## What's in here

### `VadChunker` trait (`streaming/mod.rs:44`)

`Send`-bound so a chunker can be moved into `spawn_blocking`. Object-safe (compile-time witness at `streaming/mod.rs:77`). Three methods plus an absolute-position accessor:

- `push(&mut self, samples: &[f32]) -> Vec<VadChunk>` — feed samples, get any chunks ready to dispatch.
- `flush(&mut self) -> Vec<VadChunk>` — end-of-session, surface any in-flight speech (note: returns `Vec`, not `Option`, because the zero-padded final frame can legitimately trigger both a mid-flush emission and a closing emission).
- `reset(&mut self)` — drop accumulated state.
- `next_sample_index(&self) -> u64` — what the next sample fed via `push` will be at, used by the commit-policy / buffer-trim glue.

### `RmsVadChunker` (`streaming/rms_vad.rs`)

Energy-backed VAD with hysteresis. Constants:

- `FRAME_SAMPLES = 800` (50 ms at 16 kHz).
- `DEFAULT_ENTER_RMS_THRESHOLD = 0.003`, `DEFAULT_EXIT_RMS_THRESHOLD = 0.0014` — separate thresholds give hysteresis so a score bouncing on a single value doesn't toggle state every frame.
- `DEFAULT_SPEECH_ONSET_FRAMES = 3` (3 sustained frames over enter threshold required to enter speech state — filters keyboard clicks).
- `DEFAULT_SILENCE_CLOSE_SAMPLES = 8_000` (500 ms of sub-threshold to close).
- `DEFAULT_MAX_CHUNK_SAMPLES = 32_000` (2 s, hard cap so Whisper isn't fed a 30-second monolith).
- `DEFAULT_SAMPLE_RATE_HZ = 16_000`.

Internal state machine (`streaming/rms_vad.rs:46`): `Idle` ↔ `InSpeech`. The `onset_buffer` keeps a rolling `speech_onset_frames * FRAME_SAMPLES` worth of audio so when speech is confirmed, the emitted chunk includes the onset itself, not just the post-onset audio.

Two emit paths:

- `emit_active_chunk_and_close` (`streaming/rms_vad.rs:217`) — end-of-utterance. Trims trailing silence, returns to `Idle`.
- `emit_active_chunk_continue` (`streaming/rms_vad.rs:248`) — hit `max_chunk_samples` mid-speech. Stays in `InSpeech`, advances `active_chunk_start` by the emitted length so the next chunk's start sample is contiguous.

`flush` is unusually careful (`streaming/rms_vad.rs:294`): it pads any sub-frame `pending` buffer to a full frame with zeros, runs `consume_frame` on it (which may emit a chunk), then if state is still `InSpeech` emits the active chunk as a closing chunk. Both emissions are returned. A 2026-04-22 audit (CRITICAL C2) found the prior code dropped the consume_frame chunk via `let _ = consume_frame(...)`; regression tests at `streaming/rms_vad.rs:567` and `:614` keep this honest.

### `LocalAgreement` (`streaming/commit_policy.rs`)

Source: ufal/whisper_streaming. Algorithm:

- Hold the last `n` ASR passes (default `n = 2`).
- The longest common prefix of those passes is the "agreed" prefix.
- Any tokens beyond that prefix are tentative (UI dashed-underline per workstream-B).
- Commit only grows: once a token is committed, even a divergent later pass cannot uncommit it (`new_committed = lcp_len.max(self.committed_count)`).

`Token` equality is **text-only** (`streaming/commit_policy.rs:34`); timestamps drift slightly between overlapping Whisper windows. Start/end seconds are absolute (session-relative) so the buffer-trim layer can compute sample indices.

`CommitDecision` has two vecs:

- `newly_committed` — append to the frontend's committed list.
- `tentative` — replaces (not appends to) the previous tentative list.

`flush` (`streaming/commit_policy.rs:163`) is end-of-stream: anything still tentative in the latest pass is committed and returned. Updates `last_committed_end_secs` so the buffer trim works on the closing region.

Defensive bookkeeping at `streaming/commit_policy.rs:131`: a later pass can legitimately arrive **shorter** than `committed_count` (Whisper re-transcribing an overlapping window with fewer segments). All slice indices are clamped against `latest.len()` to avoid panics. Regression test at `streaming/commit_policy.rs:317`.

### Buffer trim (`streaming/buffer_trim.rs`)

`sample_index_for_seconds(end_secs, sample_rate) -> u64`:

- Returns `0` for non-finite or `<= 0` inputs (defensive — without `is_finite`, `f64::INFINITY` would saturate to `u64::MAX` and trim the whole buffer forever).
- `(end_secs * sample_rate as f64).round() as u64` otherwise.

`trim_buffer_to_commit_point(buffer, buffer_start_sample, commit_sample_index) -> u64`:

- If the commit point is at or before the current buffer origin, do nothing.
- If it's past the buffer end, clear the buffer and park the new origin at the commit point.
- Otherwise drain the prefix and return the new origin.

The integrated property test at `streaming/buffer_trim.rs:144` simulates 100 cycles of 16 kHz captures and proves the buffer envelope stays bounded by `2 * tentative_per_cycle` instead of growing linearly.

## Data flow

```
StreamingResampler 16 kHz mono
   └─ capture_buffer (Vec<f32>, anchored at buffer_start_sample)
       └─ RmsVadChunker.push(samples) → Vec<VadChunk>
           └─ for each VadChunk:
                 spawn_blocking → LocalEngine.transcribe_sync(chunk)
                     → Vec<Segment>
                         → tokens (Token { text, start_secs, end_secs })
                             → LocalAgreement.push(tokens) → CommitDecision
                                 ├─ newly_committed → emit to frontend
                                 │   └─ LocalAgreement.last_committed_end_secs
                                 │       → sample_index_for_seconds
                                 │           → trim_buffer_to_commit_point(capture_buffer)
                                 └─ tentative → replace tentative tail in frontend
…session ends…
   └─ RmsVadChunker.flush() → final Vec<VadChunk>
   └─ LocalAgreement.flush() → final tentative committed
```

## Watch-outs

- **Not yet wired into `live.rs`.** All three primitives are unit-tested but their integration into `src-tauri/src/commands/live.rs` is explicitly described as a follow-up. Slice 2 owns the verification of whether that has landed.
- **Token equality is text-only.** Two passes that produce the same word with different casing or punctuation will not match. Whisper is fairly stable here, but a model swap is worth watching.
- **`speech_onset_frames` is a transient filter, not a denoiser.** Sustained background talk above the enter threshold will register as speech.
- **`FRAME_SAMPLES = 800` assumes 16 kHz.** A future backend at 24 kHz would need a different frame size to hit the same 50 ms window. `RmsVadChunker::sample_rate_hz` returns the constant; do not lie to it.
- **`reset()` zeros `next_sample_index`** but `flush()` deliberately does not (so a flush at end-of-session leaves the running counter accurate for any post-flush diagnostic).
- **`max_chunk_samples` is a hard 2 s cap.** A user reading aloud without a breath crosses it constantly. The continue-emit logic preserves audio contiguity (regression test at `streaming/rms_vad.rs:476`), but the resulting chunk boundaries land mid-word, which the `LocalAgreement` two-pass overlap can stitch back together — *if* the live session is feeding overlapping windows. If `live.rs` ever feeds non-overlapping chunks, expect words sliced in half.
- **`LocalAgreement::flush` may double-commit.** Calling it twice on the same instance: the second call would see `latest.len() == committed_count` and return empty. The `flush_with_no_history_is_empty` test pins this. Callers should still only call once at session end.
- **Buffer-trim's `sample_index_for_seconds` rounds nearest.** Integer comparisons against the trimmed buffer must not assume floor.

## See also

- [Audio capture pipeline](audio-capture-pipeline.md) — produces the chunks fed into `StreamingResampler`.
- [Audio resampling](audio-resampling.md) — `StreamingResampler` upstream of the chunker.
- [Transcription engines overview](transcription-engines-overview.md) — `LocalEngine.transcribe_sync` is what the streaming layer calls per chunk.
- [Audio VAD](audio-vad.md) — the deferred Silero path that will eventually replace `RmsVadChunker`.
- `docs/whisper-ecosystem/workstream-A.md`, `workstream-B.md` — the design source for thresholds and commit/tentative UX.
- Tests in `streaming/rms_vad.rs:359`, `streaming/commit_policy.rs:212`, `streaming/buffer_trim.rs:57`.