Two new Settings → AI knobs that compose cleanly with what already shipped (aiTier, LLM model, translator prompt framing). **B.1 #15 — Named cleanup presets.** LlmPromptPreset enum (Default / Email / Notes / Code) appends a short context hint onto the CLEANUP_PROMPT just before generation. Presets shape tone and structure ("email paragraph", "bulleted meeting notes", "preserve technical terms") without licensing the content-editing the translator-not-editor framing forbids. cleanup_transcript_text_cmd now takes `preset: Option<String>` which runs through the new LlmPromptPreset::parse (normalises aliases like "meeting-notes", collapses unknown values to Default). **A.1 #28 — Sequential-GPU guard.** New LocalEngine::unload drops the backend + model_id so a subsequent load actually reclaims VRAM. load_llm_model, load_model, and load_parakeet_model Tauri commands grow an optional `concurrent: bool` argument. When concurrent is Some(false), loading LLM first unloads whisper+parakeet, and vice versa — prevents VRAM OOM on tight-VRAM setups. Default is the previous parallel behaviour so nothing changes for multi-GB cards. Transcribe-in-progress paths (transcribe_pcm, transcribe_file, live) pass None, so mid-dictation model loads don't accidentally tear down the LLM. Settings UI (AI section): - Cleanup preset segmented button + descriptive copy for each option. - GPU concurrency segmented button with explicit trade-off text ("faster transitions vs fits in tight VRAM"). Frontend wiring: - settings.llmPromptPreset flows from DictationPage's cleanupTranscriptIfEnabled into the Tauri command. - settings.aiGpuConcurrency flows from both DictationPage (auto-load on record) and SettingsPage (manual load/unload buttons) as `concurrent: "parallel" === true` to the load commands. Tests: three new preset cases in crates/ai-formatting/src/llm_client.rs (parse aliases, suffix non-empty for non-default, default suffix empty). All 139 existing lib tests still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -239,7 +239,13 @@
|
||||
try {
|
||||
const status = await invoke("check_llm_model", { modelId: settings.llmModelId });
|
||||
if (status?.downloaded && !status.loaded) {
|
||||
await invoke("load_llm_model", { modelId: settings.llmModelId });
|
||||
await invoke("load_llm_model", {
|
||||
modelId: settings.llmModelId,
|
||||
// Sequential-GPU guard (brief item A.1 #28): frees whisper
|
||||
// before loading the LLM on tight-VRAM setups. "parallel"
|
||||
// keeps both resident (default, safe on multi-GB cards).
|
||||
concurrent: settings.aiGpuConcurrency === "parallel",
|
||||
});
|
||||
}
|
||||
} catch (err) {
|
||||
console.warn("ensureLlmModelLoaded failed", err);
|
||||
@@ -257,10 +263,15 @@
|
||||
error = "";
|
||||
|
||||
try {
|
||||
// concurrent flag mirrors settings.aiGpuConcurrency (brief A.1 #28).
|
||||
const concurrent = settings.aiGpuConcurrency === "parallel";
|
||||
if (settings.engine === "parakeet") {
|
||||
await invoke("load_parakeet_model", { name: "ctc-int8" });
|
||||
await invoke("load_parakeet_model", { name: "ctc-int8", concurrent });
|
||||
} else {
|
||||
await invoke("load_model", { size: settings.modelSize.toLowerCase() });
|
||||
await invoke("load_model", {
|
||||
size: settings.modelSize.toLowerCase(),
|
||||
concurrent,
|
||||
});
|
||||
}
|
||||
await refreshRuntimeCapabilities();
|
||||
modelReady = true;
|
||||
@@ -460,6 +471,7 @@
|
||||
const cleaned = await invoke("cleanup_transcript_text_cmd", {
|
||||
transcript: text,
|
||||
profileId: profilesStore.activeProfileId,
|
||||
preset: settings.llmPromptPreset,
|
||||
});
|
||||
markGenerationDone(true);
|
||||
return cleaned?.trim() ? cleaned.trim() : text;
|
||||
|
||||
@@ -531,7 +531,11 @@
|
||||
const modelId = selectedLlmModelId();
|
||||
llmStatus = "Loading...";
|
||||
try {
|
||||
await invoke("load_llm_model", { modelId });
|
||||
await invoke("load_llm_model", {
|
||||
modelId,
|
||||
// Sequential-GPU guard (brief item A.1 #28).
|
||||
concurrent: settings.aiGpuConcurrency === "parallel",
|
||||
});
|
||||
await refreshLlmStatus();
|
||||
await refreshGlobalLlmStatus(settings.aiTier);
|
||||
} catch (err) {
|
||||
@@ -740,7 +744,10 @@
|
||||
engineStatus = "Loading...";
|
||||
engineOk = false;
|
||||
try {
|
||||
await invoke("load_model", { size });
|
||||
await invoke("load_model", {
|
||||
size,
|
||||
concurrent: settings.aiGpuConcurrency === "parallel",
|
||||
});
|
||||
await refreshRuntimeCapabilities();
|
||||
engineOk = true;
|
||||
engineStatus = `${settings.modelSize} model loaded`;
|
||||
@@ -796,7 +803,10 @@
|
||||
parakeetStatus = "Loading...";
|
||||
parakeetOk = false;
|
||||
try {
|
||||
await invoke("load_parakeet_model", { name: "ctc-int8" });
|
||||
await invoke("load_parakeet_model", {
|
||||
name: "ctc-int8",
|
||||
concurrent: settings.aiGpuConcurrency === "parallel",
|
||||
});
|
||||
await refreshRuntimeCapabilities();
|
||||
parakeetOk = true;
|
||||
parakeetStatus = "Model loaded";
|
||||
@@ -1505,6 +1515,48 @@
|
||||
{/if}
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<!-- Cleanup preset (brief item B.1 #15). Shapes the tone /
|
||||
structure of LLM cleanup output; composes on top of the
|
||||
active profile's initial prompt. -->
|
||||
<div class="mt-5">
|
||||
<p class="text-[10px] font-medium text-text-tertiary uppercase tracking-wider mb-2">Cleanup preset</p>
|
||||
<SegmentedButton
|
||||
options={["default", "email", "notes", "code"]}
|
||||
bind:value={settings.llmPromptPreset}
|
||||
/>
|
||||
<p class="text-[11px] text-text-tertiary mt-2">
|
||||
{#if settings.llmPromptPreset === "email"}
|
||||
Formats output as an email paragraph — tight sentences, no markdown, no auto-added greeting or signoff.
|
||||
{:else if settings.llmPromptPreset === "notes"}
|
||||
Formats action items as a bullet list led by imperative verbs; keeps prose informational sentences as prose.
|
||||
{:else if settings.llmPromptPreset === "code"}
|
||||
Preserves technical terms, variable names, file paths, and symbols exactly as spoken. No translation of identifiers.
|
||||
{:else}
|
||||
No preset — the active profile's prompt governs tone alone.
|
||||
{/if}
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<!-- Sequential-GPU guard (brief item A.1 #28). On a tight-
|
||||
VRAM single-GPU system, loading LLM + Whisper together
|
||||
can OOM. Sequential mode unloads one before loading the
|
||||
other; adds reload latency between transcribe and
|
||||
cleanup phases. -->
|
||||
<div class="mt-5">
|
||||
<p class="text-[10px] font-medium text-text-tertiary uppercase tracking-wider mb-2">GPU concurrency</p>
|
||||
<SegmentedButton
|
||||
options={["parallel", "sequential"]}
|
||||
bind:value={settings.aiGpuConcurrency}
|
||||
/>
|
||||
<p class="text-[11px] text-text-tertiary mt-2">
|
||||
{#if settings.aiGpuConcurrency === "sequential"}
|
||||
On tight-VRAM cards (≤6 GB), loading Whisper + LLM together OOMs. Sequential mode frees the other model before loading; adds a small reload pause between transcribe and cleanup.
|
||||
{:else}
|
||||
Both models stay resident in GPU memory. Faster transitions, but needs enough VRAM to hold both at once.
|
||||
{/if}
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
{/if}
|
||||
</div>
|
||||
|
||||
@@ -61,6 +61,8 @@ const defaults: SettingsState = {
|
||||
fontSize: 14,
|
||||
aiTier: "cleanup",
|
||||
llmModelId: null,
|
||||
llmPromptPreset: "default",
|
||||
aiGpuConcurrency: "parallel",
|
||||
saveAudio: false,
|
||||
outputFolder: "",
|
||||
globalHotkey: "Ctrl+Shift+R",
|
||||
|
||||
@@ -12,6 +12,8 @@ export type WhisperModelSize =
|
||||
| "Distil-L";
|
||||
export type AiTier = "off" | "cleanup" | "tasks";
|
||||
export type LlmModelIdStr = "qwen3_1_7b" | "qwen3_4b_instruct_2507" | "qwen3_14b";
|
||||
export type LlmPromptPreset = "default" | "email" | "notes" | "code";
|
||||
export type AiGpuConcurrency = "parallel" | "sequential";
|
||||
export type TaskBucket = "inbox" | "today" | "soon" | "later";
|
||||
export type ToastSeverity = "info" | "success" | "warn" | "error";
|
||||
|
||||
@@ -47,6 +49,8 @@ export interface SettingsState {
|
||||
fontSize: number;
|
||||
aiTier: AiTier;
|
||||
llmModelId: LlmModelIdStr | null;
|
||||
llmPromptPreset: LlmPromptPreset;
|
||||
aiGpuConcurrency: AiGpuConcurrency;
|
||||
saveAudio: boolean;
|
||||
outputFolder: string;
|
||||
globalHotkey: string;
|
||||
|
||||
Reference in New Issue
Block a user