Lumotia/docs/architecture-map/05-core-storage-hotkey-build/storage-fts5-search.md

name, type, slice, last_verified

name	type	slice	last_verified
Storage FTS5 transcript search	architecture-map-page	05-core-storage-hotkey-build	2026/05/09

Storage FTS5 transcript search

Where you are: Architecture map → Core, Storage, Hotkey, Build → Storage FTS5 transcript search

Plain English summary. Full-text search over transcripts. SQLite's built-in FTS5 module gives us tokenisation, stemming, and BM25 ranking for free. The transcripts_fts virtual table is kept in lock-step with transcripts by three triggers; search_transcripts is the only public read path.

At a glance

• Function: search_transcripts(pool, query, limit) -> Result<Vec<TranscriptRow>> at crates/storage/src/database.rs:270.
• Schema: virtual table + three triggers, all created by migration v2 and rebuilt in migration v9 (table rebuild).
• Indexed columns: text, title. Other columns (engine, model, language) are not searchable today.
• Tokenizer: porter unicode61 remove_diacritics 2 — Porter stemmer, Unicode-normalised, diacritics stripped.
• Consumers: slice 2 history command (the search box on the History page).

Schema

Created in migration v2, rebuilt in migration v9 against the new (FK-bearing) transcripts table:

CREATE VIRTUAL TABLE transcripts_fts USING fts5(
    text, title,
    content='transcripts',
    content_rowid='rowid',
    tokenize='porter unicode61 remove_diacritics 2'
);

CREATE TRIGGER transcripts_ai AFTER INSERT ON transcripts BEGIN
    INSERT INTO transcripts_fts(rowid, text, title)
    VALUES (new.rowid, new.text, COALESCE(new.title, ''));
END;

CREATE TRIGGER transcripts_ad AFTER DELETE ON transcripts BEGIN
    INSERT INTO transcripts_fts(transcripts_fts, rowid, text, title)
    VALUES ('delete', old.rowid, old.text, COALESCE(old.title, ''));
END;

CREATE TRIGGER transcripts_au AFTER UPDATE ON transcripts BEGIN
    INSERT INTO transcripts_fts(transcripts_fts, rowid, text, title)
    VALUES ('delete', old.rowid, old.text, COALESCE(old.title, ''));
    INSERT INTO transcripts_fts(rowid, text, title)
    VALUES (new.rowid, new.text, COALESCE(new.title, ''));
END;

content='transcripts' makes transcripts_fts a "contentless" external-content index — the FTS table stores tokens but not the data itself, which lives in transcripts. This halves on-disk size compared with a self-contained FTS5 table.

Query shape — crates/storage/src/database.rs:275

SELECT t.<all columns>
FROM transcripts t
JOIN transcripts_fts fts ON fts.rowid = t.rowid
WHERE transcripts_fts MATCH ?
ORDER BY fts.rank
LIMIT ?

fts.rank is FTS5's negative BM25 score (most-relevant first). MATCH accepts standard FTS5 syntax: bare words AND together; OR, NOT, parentheses; quoted multi-word phrases; column-scoped queries (text:foo title:bar).

Watch-outs

• MATCH requires the leading word to be a real query, not a malformed token. A user typing a single quote into the search box will produce a SQL: malformed match error from sqlite. The frontend should sanitise or fall back; today it does not, so the error propagates to the toast.
• The triggers fire on every transcript insert / delete / update. A bulk import of 1,000 transcripts walks the FTS path 1,000 times. Acceptable today (transcripts are user-authored, not imported). If bulk-import lands, consider a transcripts_fts(transcripts_fts) VALUES('rebuild') after the bulk insert.
• Rebuild on schema change is expensive. Migration v9 rebuilt the FTS table and re-tokenised every existing transcript. On a heavy user's database this can take several seconds. v9 ran inside a transaction so a failed rebuild rolls back cleanly.
• title is COALESCE-d to ''. A null title becomes an empty FTS document, not a missing column.
• No FTS over manual_tags or llm_tags. Those columns hold comma-joined values and are filtered with LIKE queries today. A future migration could add them to the FTS5 index.

Tests

No dedicated FTS test in database.rs. Coverage is via the migration tests (which assert the table and triggers exist post-migration) and via the integration test pattern tx.execute("INSERT INTO transcripts ..."); search_transcripts("phrase") used in the slice-2 command tests.

Worth adding a unit test that asserts BM25 ranking actually picks the better match — today a regression in the tokenize= clause would not break compile or migration tests.

See also

• Storage overview
• Schema and migrations — v2 and v9 are the relevant points.
• Transcripts CRUD