44 lines
2.0 KiB
Markdown
44 lines
2.0 KiB
Markdown
# RB-10 MAJOR: LLM prompts not preflighted against context window
|
||
|
||
**Severity:** MAJOR
|
||
**Path:** `crates/llm/src/lib.rs:143-166`, `:317-321`
|
||
**Source:** [2026-04-22 code review](../code-review-2026-04-22.md)
|
||
**Labels:** release-blocker, major, llm
|
||
**Status:** RESOLVED (2026-04-22)
|
||
|
||
## Resolution
|
||
|
||
`LlmEngine::generate` still tokenises the whole prompt up front, but it
|
||
now runs a dedicated prompt-budget preflight before creating the llama
|
||
context. The chosen behaviour is an early typed failure rather than
|
||
silent truncation:
|
||
|
||
- If `prompt_tokens + max_tokens + 64 reserve tokens` exceeds the
|
||
8192-token cap, generation returns
|
||
`EngineError::PromptTooLong { prompt_tokens, max_tokens, available_prompt_tokens, context_window }`.
|
||
- Prompts that fit exactly within the available budget still proceed and
|
||
allocate an 8192-token context as before.
|
||
|
||
Regression tests:
|
||
|
||
- `prompt_preflight_rejects_oversized_prompt_tokens`
|
||
- `prompt_preflight_keeps_prompts_within_budget`
|
||
|
||
## Problem
|
||
|
||
`generate` tokenises and batches the full prompt at runtime. `context_window_size` hard-caps context at 8192 tokens. Long transcripts (a 30-minute dictation session is easily 4000–6000 tokens after segment joining) reach inference with prompts already bigger than the available context — causing late runtime failure instead of a controlled early-exit path.
|
||
|
||
## Acceptance
|
||
|
||
- Before inference begins, the prompt token count is compared against the available context window (minus the expected response budget).
|
||
- Oversized prompts either (a) surface a typed error the caller can handle gracefully, or (b) are truncated with a logged warning — decide during the fix.
|
||
- Regression test: synthesise a transcript whose tokenised form exceeds 8192 tokens, assert the chosen behaviour (early error or truncated input).
|
||
|
||
## Fix scope
|
||
|
||
Medium. Tokeniser access is already on the LLM path; the check is cheap. Decision work is in what to do when a prompt is too long (fail hard vs truncate).
|
||
|
||
## Dependencies
|
||
|
||
- None — standalone fix.
|