Lumotia/docs/architecture-map/05-core-storage-hotkey-build/storage-schema-and-migrations.md

name, type, slice, last_verified

name	type	slice	last_verified
Storage schema and migrations	architecture-map-page	05-core-storage-hotkey-build	2026/05/09

Storage schema and migrations

Where you are: Architecture map → Core, Storage, Hotkey, Build → Storage schema and migrations

Plain English summary. The full database schema as currently shipped, plus the append-only history of migrations that brought it to v15. Each migration is one transaction. If a migration fails partway, the transaction rolls back so the schema never ends up half-applied. The schema is forward-only — there are no down-migrations.

At a glance

• File: crates/storage/src/migrations.rs (1,185 LOC).
• External deps: sqlx 0.8 (SqlitePool), magnotia_core::error.
• Public surface: run_migrations(pool: &SqlitePool) -> Result<()>. That is the only pub symbol.
• Schema head: v15.
• Consumers: called by magnotia_storage::database::init at crates/storage/src/database.rs:54.

Migration runner contract

The migration registry is a &[(i64, &str, &str)] triple: (version, description, sql). The runner is at crates/storage/src/migrations.rs:525:

pub async fn run_migrations(pool: &SqlitePool) -> Result<()> {
    run_migrations_slice(pool, MIGRATIONS).await
}

run_migrations_slice (migrations.rs:546):

1. Ensures schema_version (version INTEGER PRIMARY KEY, description TEXT NOT NULL, applied_at TEXT NOT NULL DEFAULT datetime('now')) exists.
2. Reads MAX(version) from schema_version.
3. For each (version, description, sql) whose version > current:
   • Begin a tx.
   • Split the sql blob into individual statements via split_statements (a hand-rolled SQLite-trigger-aware splitter at migrations.rs:483).
   • Execute each statement against the transaction.
   • Insert the version row into schema_version inside the same transaction.
   • Commit.
4. Idempotent: re-running on a head schema does nothing.

Atomicity (one tx per migration) was added in response to the 2026-04-22 review (RB-02 / C3). Documented inline at migrations.rs:548-565. Constraint: future migrations must avoid SQLite operations that implicitly commit (VACUUM, REINDEX, ATTACH). If one is needed, split it into its own non-transactional migration — reviewer's job to flag.

Schema head (v15)

The shape of every table at head, derived by replaying v1 → v15.

transcripts — keyed id TEXT PRIMARY KEY

CREATE TABLE transcripts (
    id TEXT PRIMARY KEY,
    text TEXT NOT NULL DEFAULT '',
    source TEXT NOT NULL DEFAULT 'microphone',
    title TEXT,
    audio_path TEXT,
    duration REAL NOT NULL DEFAULT 0.0,
    engine TEXT,
    model_id TEXT,
    inference_ms INTEGER,
    sample_rate INTEGER,
    audio_channels INTEGER,
    format_mode TEXT,
    remove_fillers INTEGER NOT NULL DEFAULT 0,
    british_english INTEGER NOT NULL DEFAULT 0,
    anti_hallucination INTEGER NOT NULL DEFAULT 0,
    created_at TEXT NOT NULL DEFAULT (datetime('now')),
    -- v5 (transcripts_meta):
    starred INTEGER NOT NULL DEFAULT 0,
    manual_tags TEXT NOT NULL DEFAULT '',
    template TEXT NOT NULL DEFAULT '',
    language TEXT NOT NULL DEFAULT '',
    segments_json TEXT NOT NULL DEFAULT '',
    -- v8/v9 (profile FK):
    profile_id TEXT NOT NULL REFERENCES profiles(id),
    -- v14:
    llm_tags TEXT NOT NULL DEFAULT ''
);

CREATE INDEX idx_transcripts_created ON transcripts(created_at);
CREATE INDEX idx_transcripts_profile_id ON transcripts(profile_id);
CREATE INDEX idx_transcripts_profile_created ON transcripts(profile_id, created_at DESC);  -- v15

segments — keyed INTEGER PRIMARY KEY AUTOINCREMENT

CREATE TABLE segments (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    transcript_id TEXT NOT NULL REFERENCES transcripts(id) ON DELETE CASCADE,
    start_time REAL NOT NULL,
    end_time REAL NOT NULL,
    text TEXT NOT NULL DEFAULT ''
);
CREATE INDEX idx_segments_transcript ON segments(transcript_id);

(Note: today the engines store full segments in transcripts.segments_json rather than this table. The table is kept for backwards compatibility.)

transcripts_fts — FTS5 virtual table (v2, rebuilt v9)

CREATE VIRTUAL TABLE transcripts_fts USING fts5(
    text, title,
    content='transcripts', content_rowid='rowid',
    tokenize='porter unicode61 remove_diacritics 2'
);

Plus three triggers (transcripts_ai, transcripts_ad, transcripts_au) that mirror inserts / deletes / updates from transcripts into the FTS index. See storage-fts5-search.md.

tasks

CREATE TABLE tasks (
    id TEXT PRIMARY KEY,
    text TEXT NOT NULL,
    bucket TEXT NOT NULL DEFAULT 'inbox',
    list_id TEXT,
    effort TEXT,
    done INTEGER NOT NULL DEFAULT 0,
    done_at TEXT,
    created_at TEXT NOT NULL DEFAULT (datetime('now')),
    source_transcript_id TEXT,
    -- v3:
    parent_task_id TEXT REFERENCES tasks(id) ON DELETE CASCADE,
    -- v4:
    notes TEXT NOT NULL DEFAULT '',
    -- v11:
    energy TEXT,  -- CHECK (energy IN ('high', 'medium', 'brain_dead') OR energy IS NULL)
    -- v13:
    auto_completed INTEGER NOT NULL DEFAULT 0
);

CREATE INDEX idx_tasks_bucket ON tasks(bucket);
CREATE INDEX idx_tasks_transcript ON tasks(source_transcript_id);
CREATE INDEX idx_tasks_parent ON tasks(parent_task_id);
CREATE INDEX idx_tasks_energy_created ON tasks(energy, created_at);  -- v11
CREATE INDEX idx_tasks_done_at_auto_completed ON tasks(done_at, auto_completed);  -- v13

task_lists

CREATE TABLE task_lists (
    id TEXT PRIMARY KEY,
    name TEXT NOT NULL,
    built_in INTEGER NOT NULL DEFAULT 0,
    profile_id TEXT,
    created_at TEXT NOT NULL DEFAULT (datetime('now'))
);

profiles (v6)

CREATE TABLE profiles (
    id TEXT PRIMARY KEY,
    name TEXT NOT NULL UNIQUE,
    initial_prompt TEXT NOT NULL DEFAULT '',
    created_at TEXT NOT NULL DEFAULT (datetime('now'))
);

Default profile seeded at id 00000000-0000-0000-0000-000000000001. Two SQL triggers protect it:

CREATE TRIGGER trg_protect_default_profile_delete
    BEFORE DELETE ON profiles
    WHEN OLD.id = '00000000-0000-0000-0000-000000000001'
    BEGIN SELECT RAISE(ABORT, 'cannot delete the default profile'); END;

CREATE TRIGGER trg_protect_default_profile_rename
    BEFORE UPDATE OF id, name ON profiles
    WHEN OLD.id = '00000000-0000-0000-0000-000000000001'
    BEGIN SELECT RAISE(ABORT, 'cannot rename the default profile'); END;

profile_terms (v6)

CREATE TABLE profile_terms (
    id TEXT PRIMARY KEY,
    profile_id TEXT NOT NULL REFERENCES profiles(id) ON DELETE CASCADE,
    term TEXT NOT NULL,
    note TEXT,
    created_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE INDEX idx_profile_terms_profile_id ON profile_terms(profile_id);

UUIDs for id are generated as v4 random per crates/storage/Cargo.toml. (The Cargo.toml comment says "v7 random" but the feature flag is ["v4"]. This is a known inconsistency between comment and reality. The code path uses Uuid::new_v4().)

settings

Plain key-value store. key TEXT PRIMARY KEY, value TEXT NOT NULL. The frontend's preferences blob lives at key magnotia_preferences.

error_log

Append-only diagnostic log. Pruned by prune_error_log on app startup with a 90-day default. See storage-crud-settings-and-misc.md.

feedback (v10)

HITL thumbs + corrections, scoped to profile. See storage-crud-settings-and-misc.md.

implementation_rules (v12)

If-then automation rules. See storage-crud-settings-and-misc.md.

schema_version

Bookkeeping. Created by the migration runner itself.

Migration history

15 migrations, all in MIGRATIONS: &[(i64, &str, &str)] at crates/storage/src/migrations.rs:7.

v	Description	What changes
1	initial schema	Creates transcripts, segments, tasks, task_lists, settings, error_log plus 5 base indexes.
2	transcripts FTS5 + dictionary table	Creates transcripts_fts virtual table, three sync triggers, dictionary table (later removed).
3	micro-stepping: parent_task_id on tasks	ALTER tasks ADD COLUMN parent_task_id (with FK + cascade); idx_tasks_parent.
4	tasks_meta: notes column	ALTER tasks ADD COLUMN notes TEXT NOT NULL DEFAULT ''.
5	transcripts_meta	Adds starred, manual_tags, template, language, segments_json columns. Persists what was previously in localStorage magnotia_history.
6	profiles	Creates profiles and profile_terms tables; seeds the default profile; copies dictionary rows to the default profile's profile_terms; installs the two protection triggers.
7	drop_dictionary	Removes the legacy dictionary table and its index. Data has already been copied in v6.
8	transcript_profile_provenance	Adds transcripts.profile_id (nullable initially); idx_transcripts_profile_id.
9	transcript_profile_fk	Table rebuild. Reconciles orphaned transcripts.profile_id values to the default profile, then rebuilds transcripts and segments with the FK declared. The FTS table and triggers are dropped and recreated against the rebuilt transcripts. Driven by docs/issues/c4-transcript-profile-fk.md.
10	feedback: HITL thumbs + correction capture	Creates feedback table; two indexes (target_type+rating, profile).
11	tasks: energy tagging	ALTER tasks ADD COLUMN energy with CHECK constraint; idx_tasks_energy_created.
12	implementation intentions	Creates implementation_rules table; idx_implementation_rules_enabled_trigger.
13	gamification: auto_completed	ALTER tasks ADD COLUMN auto_completed; idx_tasks_done_at_auto_completed. Cascade-completed parents are tagged so the daily-completion query can exclude them.
14	transcripts: llm_tags column	ALTER transcripts ADD COLUMN llm_tags TEXT NOT NULL DEFAULT ''. Phase 9 LLM content tags.
15	composite (profile_id, created_at DESC) index	CREATE INDEX idx_transcripts_profile_created. Speeds up the profile-scoped history page query that currently uses idx_transcripts_profile_id alone.

Tests

Migration tests live alongside the registry at crates/storage/src/migrations.rs:606-1184:

• test_migrations_run_on_empty_db — full v1→head replay.
• test_migrations_idempotent — running twice is a no-op.
• migration_tasks_meta_adds_columns — v3 ALTER goes through.
• migration_implementation_rules_adds_rule_table — v12 happy path.
• migration_transcripts_meta_adds_columns — v5 ALTER goes through.
• migration_transcript_profile_provenance_adds_profile_id — v8.
• migration_v9_reconciles_orphaned_transcript_profiles_and_adds_fk — the v9 table rebuild end-to-end.
• test_parent_task_id_cascade_delete — v3 cascade FK behaviour.
• migration_v6_seeds_default_profile_on_fresh_db.
• migration_v6_copies_dictionary_rows_to_default_profile_terms.
• migration_v6_trigger_rejects_default_profile_delete.
• migration_v6_trigger_rejects_default_profile_rename.
• migration_v7_drops_dictionary_table.
• multi_statement_migration_rolls_back_on_failure — the C3 atomicity guarantee.
• migration_v13_adds_auto_completed_column.
• migration_v15_creates_profile_created_index — the head migration.

Watch-outs

• Append-only. Modifying an existing migration after release is a contract violation. Add a new one.
• split_statements is hand-rolled. Triggers contain BEGIN ... END with embedded semicolons. The splitter at migrations.rs:483 is trigger-aware (it tracks BEGIN / END depth) but the parser is hand-rolled rather than a real SQL tokeniser. Adding a new control-flow keyword (eg CASE WHEN ... END outside a trigger) could trip it. Test additions are advised.
• Migration v9 is the only table rebuild. Rebuilds drop the FTS index and recreate it. If a future column needs FK adjustment, the v9 pattern is the template, but it is more risk than an ALTER.
• No down migrations. Recovery from a bad migration is restore-from-backup. The pre-commit pattern is to test on a fresh DB and against a copy of a real one.

Existing in-repo docs

• docs/issues/c3-migrations-atomicity.md — RB-02. Drove the per-migration transaction wrapper.
• docs/issues/c4-transcript-profile-fk.md — drove migration v9.

See also

• Storage overview
• FTS5 search
• Transcripts CRUD
• Tasks CRUD
• Profiles CRUD
• Settings, error log, feedback, implementation rules