Lumotia/docs/architecture-map/03-audio-transcription/transcription-concurrency.md

name, type, slice, last_verified

name	type	slice	last_verified
Inference concurrency (run_inference)	architecture-map-page	03-audio-transcription	2026/05/09

Inference concurrency

Where you are: Architecture map → Audio + Transcription → Inference concurrency

Plain English summary. Whisper and Parakeet inference is synchronous and CPU-bound (or GPU-bound through Vulkan). Running it on the Tokio runtime would block the executor, so every async caller goes through run_inference, a thin wrapper that hands the work to tokio::task::spawn_blocking. Callers never see the Arc<LocalEngine> lock or the spawn boundary; they await a TimedTranscript.

At a glance

• Crate: lumotia-transcription
• Path: crates/transcription/src/concurrency.rs
• LOC: 19
• External deps: tokio (rt feature for spawn_blocking).
• Internal callers (best effort, slice 2 reconciles): src-tauri/src/commands/transcription.rs (file path), src-tauri/src/commands/live.rs (per-chunk during live capture).

Public surface:

• pub async fn run_inference(engine: Arc<LocalEngine>, audio: AudioSamples, options: TranscriptionOptions) -> Result<TimedTranscript> — crates/transcription/src/concurrency.rs:11.

What's in here

The whole file:

pub async fn run_inference(
    engine: Arc<LocalEngine>,
    audio: AudioSamples,
    options: TranscriptionOptions,
) -> Result<TimedTranscript> {
    tokio::task::spawn_blocking(move || engine.transcribe_sync(&audio, &options))
        .await
        .map_err(|e| MagnotiaError::TranscriptionFailed(format!("Task join error: {e}")))?
}

engine is shared via Arc (the engine itself uses an internal mutex for inference / load serialisation). audio and options are moved into the closure. The double ? flattens the join-error layer with the inference-error layer.

Data flow

async caller
  └─ run_inference(engine, audio, options)
      └─ tokio::task::spawn_blocking(move || engine.transcribe_sync(...))
          └─ LocalEngine internal mutex → Transcriber::transcribe_sync
              → Vec<Segment> + inference_ms
          → Result<TimedTranscript>
      .await + map_err(JoinError → MagnotiaError::TranscriptionFailed)

Watch-outs

• spawn_blocking is the only place inference runs. Anyone writing a new caller that calls LocalEngine::transcribe_sync directly from an async context will block the Tokio executor. Always go through run_inference.
• Arc<LocalEngine> is the contract. The function takes an Arc, not &LocalEngine, because the engine must outlive the spawned task. The shared state in slice 2 holds the engine inside an Arc.
• Audio and options are moved. They are not borrowed across the spawn boundary; the caller hands ownership over.
• JoinError is the only failure mode added by this layer. MagnotiaError::TranscriptionFailed("Task join error: ...") indicates a panic inside transcribe_sync or a runtime shutdown. The actual inference errors flow through the inner Result.
• No backpressure here. Callers issuing many concurrent run_inference calls each spawn an independent blocking task. Tokio's default blocking thread pool is 512 threads, but the LocalEngine mutex serialises them — only one will actually run inference at a time. Worth knowing if you ever look at thread counts in a profiler.

See also

• Transcription engines overview — LocalEngine::transcribe_sync is what runs inside the blocking task.
• Transcription streaming — live capture invokes run_inference per chunk.