---
name: Inference concurrency (run_inference)
type: architecture-map-page
slice: 03-audio-transcription
last_verified: 2026/05/09
---

# Inference concurrency

> **Where you are:** [Architecture map](../README.md) → [Audio + Transcription](README.md) → Inference concurrency

**Plain English summary.** Whisper and Parakeet inference is synchronous and CPU-bound (or GPU-bound through Vulkan). Running it on the Tokio runtime would block the executor, so every async caller goes through `run_inference`, a thin wrapper that hands the work to `tokio::task::spawn_blocking`. Callers never see the `Arc<LocalEngine>` lock or the spawn boundary; they `await` a `TimedTranscript`.

## At a glance

- Crate: `lumotia-transcription`
- Path: `crates/transcription/src/concurrency.rs`
- LOC: 19
- External deps: `tokio` (rt feature for `spawn_blocking`).
- Internal callers (best effort, slice 2 reconciles): `src-tauri/src/commands/transcription.rs` (file path), `src-tauri/src/commands/live.rs` (per-chunk during live capture).

Public surface:

- `pub async fn run_inference(engine: Arc<LocalEngine>, audio: AudioSamples, options: TranscriptionOptions) -> Result<TimedTranscript>` — `crates/transcription/src/concurrency.rs:11`.

## What's in here

The whole file:

```rust
pub async fn run_inference(
    engine: Arc<LocalEngine>,
    audio: AudioSamples,
    options: TranscriptionOptions,
) -> Result<TimedTranscript> {
    tokio::task::spawn_blocking(move || engine.transcribe_sync(&audio, &options))
        .await
        .map_err(|e| MagnotiaError::TranscriptionFailed(format!("Task join error: {e}")))?
}
```

`engine` is shared via `Arc` (the engine itself uses an internal mutex for inference / load serialisation). `audio` and `options` are moved into the closure. The double `?` flattens the join-error layer with the inference-error layer.

## Data flow

```
async caller
  └─ run_inference(engine, audio, options)
      └─ tokio::task::spawn_blocking(move || engine.transcribe_sync(...))
          └─ LocalEngine internal mutex → Transcriber::transcribe_sync
              → Vec<Segment> + inference_ms
          → Result<TimedTranscript>
      .await + map_err(JoinError → MagnotiaError::TranscriptionFailed)
```

## Watch-outs

- **`spawn_blocking` is the only place inference runs.** Anyone writing a new caller that calls `LocalEngine::transcribe_sync` directly from an async context will block the Tokio executor. Always go through `run_inference`.
- **`Arc<LocalEngine>` is the contract.** The function takes an `Arc`, not `&LocalEngine`, because the engine must outlive the spawned task. The shared state in slice 2 holds the engine inside an `Arc`.
- **Audio and options are moved.** They are not borrowed across the spawn boundary; the caller hands ownership over.
- **JoinError is the only failure mode added by this layer.** `MagnotiaError::TranscriptionFailed("Task join error: ...")` indicates a panic inside `transcribe_sync` or a runtime shutdown. The actual inference errors flow through the inner `Result`.
- **No backpressure here.** Callers issuing many concurrent `run_inference` calls each spawn an independent blocking task. Tokio's default blocking thread pool is 512 threads, but the `LocalEngine` mutex serialises them — only one will actually run inference at a time. Worth knowing if you ever look at thread counts in a profiler.

## See also

- [Transcription engines overview](transcription-engines-overview.md) — `LocalEngine::transcribe_sync` is what runs inside the blocking task.
- [Transcription streaming](transcription-streaming.md) — live capture invokes `run_inference` per chunk.