Jake
d1eb56fac9
kon-llm now owns a real LlamaBackend + LlamaModel, with three Qwen3 tiers (1.7B Q4, 4B-Instruct-2507 Q4, 14B Q5) selectable per hardware. Downloads are resumable with SHA-256 verification and stored under ~/.kon/models/llm. Engine exposes three high-level surfaces — all greedy/temp-0, GBNF-constrained where output shape matters: - cleanup_text (prompt-injection-hardened system prompt; profile terms appended as "preserve these spellings" suffix) - decompose_task (3–7 micro-steps, constrained JSON array) - extract_tasks (optional-array; empty when no explicit commitments) post_process_segments now takes an Option<&LlmEngine> and, when loaded and format_mode != Raw, joins segments → cleanup → replaces segments with the cleaned text (first segment span). Rule-based path still runs first; LLM errors log and keep rule-based output. Tauri commands: recommend_llm_tier, check_llm_model, download_llm_model, load_llm_model, unload_llm_model, delete_llm_model, get_llm_status, cleanup_transcript_text_cmd, extract_tasks_from_transcript_cmd, decompose_and_store (LLM-backed subtasks). Settings: AI tier toggle (off / cleanup / tasks), model picker with downloaded/loaded status, download progress events via kon:llm-download-progress. Dictation: ensureLlmModelLoaded on mount, cleanupTranscriptIfEnabled after stop when tier != off and format_mode != Raw, LLM task extraction when tier=tasks (regex fallback on failure). Interim: both llama-cpp-sys-2 and whisper-rs-sys statically link their own ggml, so src-tauri/build.rs emits -Wl,--allow-multiple-definition on Linux. Replace with a system-ggml shared-lib setup as a follow-up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
63 lines
1.8 KiB
Rust
63 lines
1.8 KiB
Rust
//! Smoke test: load a GGUF model and exercise the high-level wrappers.
|
|
//!
|
|
//! Verified against llama-cpp-2 `0.1.144` using:
|
|
//! - `llama_backend::LlamaBackend`
|
|
//! - `model::LlamaModel`
|
|
//! - `context::params::LlamaContextParams`
|
|
//! - `sampling::LlamaSampler`
|
|
//!
|
|
//! The test is gated behind `KON_LLM_TEST_MODEL`.
|
|
|
|
use std::env;
|
|
use std::path::PathBuf;
|
|
|
|
use kon_llm::LlmEngine;
|
|
use kon_llm::LlmModelId;
|
|
|
|
#[test]
|
|
fn llama_cpp_2_smoke_generates_and_wraps() {
|
|
let model_path = match env::var("KON_LLM_TEST_MODEL") {
|
|
Ok(path) => PathBuf::from(path),
|
|
Err(_) => {
|
|
eprintln!("KON_LLM_TEST_MODEL not set — skipping");
|
|
return;
|
|
}
|
|
};
|
|
|
|
let engine = LlmEngine::new();
|
|
engine
|
|
.load_model(LlmModelId::Qwen3_1_7B_Q4, &model_path, true)
|
|
.expect("load model");
|
|
|
|
let completion = engine
|
|
.generate(
|
|
"Write exactly one short greeting.",
|
|
&kon_llm::GenerationConfig {
|
|
max_tokens: 32,
|
|
temperature: 0.0,
|
|
stop_sequences: vec!["\n".to_string()],
|
|
grammar: None,
|
|
},
|
|
)
|
|
.expect("generate");
|
|
assert!(!completion.trim().is_empty());
|
|
|
|
let cleaned = engine
|
|
.cleanup_text(
|
|
"You are a transcript cleanup assistant. Remove fillers and output only cleaned text.",
|
|
"um hello there like general kenobi",
|
|
)
|
|
.expect("cleanup_text");
|
|
assert!(!cleaned.trim().is_empty());
|
|
|
|
let tasks = engine
|
|
.extract_tasks("I need to call the plumber tomorrow and buy milk.")
|
|
.expect("extract_tasks");
|
|
assert!(!tasks.is_empty());
|
|
|
|
let steps = engine
|
|
.decompose_task("Plan a weekend trip to the coast")
|
|
.expect("decompose_task");
|
|
assert!((3..=7).contains(&steps.len()));
|
|
}
|