docs: KI-07 — llama-cpp-2 v0.1.146 panics on Qwen3.5+ FGDN tensors

Discovered while spinning up a Tauri dev session on v0.3 for live UI
review. The app aborts on startup the moment DictationPage mounts and
calls load_llm_model — llama.cpp's GGML_ASSERT on the Fused Gated Delta
Net tensor-name prefix fires before the WebView paints, taking the
parent process with it. All four configured model variants
(Qwen3.5 2B/4B/9B + Qwen3.6 27B) share the architecture, so picking
a different size does not work around it.

Adds a new LLM section with KI-07 (Status / Source / Impact / Workarounds).
Four ordered workarounds, cheapest first:
  1. Park the .gguf file — autoload bails, non-LLM features work
  2. Set aiTier = "off" in lumotia_preferences before launch
  3. Bump llama-cpp-2 to a release with the upstream FGDN fix
  4. Add a non-FGDN model variant to LlmModelId

(1) unblocked tonight's live review of the Phase 5f polish. (3) is the
real resolution.
This commit is contained in:
2026-05-15 20:38:01 +01:00
parent 3810fdda43
commit baab7b3c12

View File

@@ -83,6 +83,42 @@ Display sleep is intentionally NOT blocked — the user is dictating, not watchi
**Workaround:** N/A. Existing path works as before.
## LLM
### KI-07 — `llama-cpp-2 v0.1.146` panics on Qwen3.5+ Fused Gated Delta Net (FGDN) tensors
**Status:** Loading any of the four configured model variants (`Qwen3_5_2B_Q4`, `Qwen3_5_4B_Q4`, `Qwen3_5_9B_Q4`, `Qwen3_6_27B_Q4`) at startup aborts the whole process with a GGML assert inside the bundled `llama.cpp`:
```
llama-cpp-sys-2-0.1.146/llama.cpp/src/llama-context.cpp:487:
GGML_ASSERT(strncmp(n->name, LLAMA_TENSOR_NAME_FGDN_AR "-", prefix_len) == 0) failed
```
The assert fires during `llama_context` construction while resolving Fused Gated Delta Net support, which is the recurrent / Mamba-style attention mechanism Qwen3.5+ uses. `llama-cpp-2 v0.1.146` bundles a `llama.cpp` snapshot whose FGDN tensor-name prefix check rejects tensors as named in current Qwen3.5 GGUFs. Because the assert is in C++ via `ggml`, the panic kills the parent Tauri process — Tauri's crash handler then attaches GDB, which is what surfaces in the dev log as a stack trace ending in `tao::event_loop` rather than as a Rust panic.
All four enum variants share the same architecture family, so picking a different size does not work around it.
**Source:**
- Dep pin: [`crates/llm/Cargo.toml:24`](crates/llm/Cargo.toml#L24) (`llama-cpp-2 = "0.1.146"`)
- Model enum: [`crates/llm/src/model_manager.rs:15`](crates/llm/src/model_manager.rs#L15) (the four Qwen3.5+ variants)
- Frontend autoload trigger: [`src/lib/pages/DictationPage.svelte:274`](src/lib/pages/DictationPage.svelte#L274) (`ensureLlmModelLoaded``invoke("load_llm_model", …)`)
- Backend command: [`src-tauri/src/commands/llm.rs:94`](src-tauri/src/commands/llm.rs#L94) (`load_llm_model`)
- Upstream assert: `llama-cpp-sys-2-0.1.146/llama.cpp/src/llama-context.cpp:487`
**Impact:** App will not start in dev (and presumably release) on any machine that has a Qwen3.5+ `.gguf` present and `settings.aiTier !== "off"`. There is no UI to recover — the crash precedes the WebView paint, so the user cannot reach Settings to disable AI features or swap the model. Affects every code path that exercises the LLM (transcript cleanup, task extraction, micro-step decomposition).
**Workarounds (in order of cost):**
1. **Park the model file** so `check_llm_model` returns `downloaded: false` and autoload silently bails:
```
mv ~/.local/share/lumotia/models/llm/Qwen3.5-4B-Q4_K_M.gguf{,.crash-parked}
```
Non-LLM features (dictation, transcript history, file imports) work normally. Reverse with `mv …{.crash-parked,}`.
2. **Set `aiTier = "off"`** in `lumotia_preferences` before launch (also bypasses the autoload guard).
3. **Bump `llama-cpp-2`** to a release that includes the upstream `llama.cpp` FGDN fix. Track upstream PRs at <https://github.com/utilityai/llama-cpp-rs/releases>; verify the bundled `llama.cpp` SHA covers Gated Delta Net tensor-name handling before adopting.
4. **Add a non-FGDN model variant** to `LlmModelId` (e.g. Llama 3.2 3B Q4 — well-supported in 0.1.146 — or Qwen2.5 3B) and ship a Settings UI to switch to it. Larger change because the enum, the on-disk URL/size constants, and the Settings model picker all need entries.
**Resolution (deferred):** Option 3 is the real fix and matches the spirit of `crates/llm` (Qwen3.5+ explicitly listed in the crate description). Option 4 is the right fallback if upstream is slow — having one non-FGDN model on the enum prevents the dev-blocker class of crash entirely.
## How to add an entry
When shipping a partial implementation or known limitation, add a `KI-NN` entry here with the four standard fields: **Status**, **Source** (file:line), **Impact**, **Workaround**. Link from the affected module's doc comment back to this file by ID.