Lumotia/docs/issues/llm-prompt-preflight.md

RB-10 MAJOR: LLM prompts not preflighted against context window

Severity: MAJOR Path: crates/llm/src/lib.rs:143-166, :317-321 Source: 2026-04-22 code review Labels: release-blocker, major, llm Status: RESOLVED (2026-04-22)

Resolution

LlmEngine::generate still tokenises the whole prompt up front, but it now runs a dedicated prompt-budget preflight before creating the llama context. The chosen behaviour is an early typed failure rather than silent truncation:

• If prompt_tokens + max_tokens + 64 reserve tokens exceeds the 8192-token cap, generation returns EngineError::PromptTooLong { prompt_tokens, max_tokens, available_prompt_tokens, context_window }.
• Prompts that fit exactly within the available budget still proceed and allocate an 8192-token context as before.

Regression tests:

• prompt_preflight_rejects_oversized_prompt_tokens
• prompt_preflight_keeps_prompts_within_budget

Problem

generate tokenises and batches the full prompt at runtime. context_window_size hard-caps context at 8192 tokens. Long transcripts (a 30-minute dictation session is easily 4000–6000 tokens after segment joining) reach inference with prompts already bigger than the available context — causing late runtime failure instead of a controlled early-exit path.

Acceptance

• Before inference begins, the prompt token count is compared against the available context window (minus the expected response budget).
• Oversized prompts either (a) surface a typed error the caller can handle gracefully, or (b) are truncated with a logged warning — decide during the fix.
• Regression test: synthesise a transcript whose tokenised form exceeds 8192 tokens, assert the chosen behaviour (early error or truncated input).

Fix scope

Medium. Tokeniser access is already on the LLM path; the check is cheap. Decision work is in what to do when a prompt is too long (fail hard vs truncate).

Dependencies

• None — standalone fix.