---
name: Storage schema and migrations
type: architecture-map-page
slice: 05-core-storage-hotkey-build
last_verified: 2026/05/09
---

# Storage schema and migrations

> **Where you are:** [Architecture map](../README.md) → [Core, Storage, Hotkey, Build](README.md) → Storage schema and migrations

**Plain English summary.** The full database schema as currently shipped, plus the append-only history of migrations that brought it to v15. Each migration is one transaction. If a migration fails partway, the transaction rolls back so the schema never ends up half-applied. The schema is forward-only — there are no down-migrations.

## At a glance

- File: `crates/storage/src/migrations.rs` (1,185 LOC).
- External deps: `sqlx 0.8` (`SqlitePool`), `lumotia_core::error`.
- Public surface: `run_migrations(pool: &SqlitePool) -> Result<()>`. That is the only `pub` symbol.
- Schema head: **v15**.
- Consumers: called by `lumotia_storage::database::init` at `crates/storage/src/database.rs:54`.

## Migration runner contract

The migration registry is a `&[(i64, &str, &str)]` triple: `(version, description, sql)`. The runner is at `crates/storage/src/migrations.rs:525`:

```rust
pub async fn run_migrations(pool: &SqlitePool) -> Result<()> {
    run_migrations_slice(pool, MIGRATIONS).await
}
```

`run_migrations_slice` (`migrations.rs:546`):

1. Ensures `schema_version (version INTEGER PRIMARY KEY, description TEXT NOT NULL, applied_at TEXT NOT NULL DEFAULT datetime('now'))` exists.
2. Reads `MAX(version)` from `schema_version`.
3. For each `(version, description, sql)` whose `version > current`:
    - Begin a `tx`.
    - Split the `sql` blob into individual statements via `split_statements` (a hand-rolled SQLite-trigger-aware splitter at `migrations.rs:483`).
    - Execute each statement against the transaction.
    - Insert the version row into `schema_version` inside the same transaction.
    - Commit.
4. Idempotent: re-running on a head schema does nothing.

Atomicity (one tx per migration) was added in response to the 2026-04-22 review (RB-02 / C3). Documented inline at `migrations.rs:548-565`. **Constraint:** future migrations must avoid SQLite operations that implicitly commit (`VACUUM`, `REINDEX`, `ATTACH`). If one is needed, split it into its own non-transactional migration — reviewer's job to flag.

## Schema head (v15)

The shape of every table at head, derived by replaying v1 → v15.

### `transcripts` — keyed `id TEXT PRIMARY KEY`

```sql
CREATE TABLE transcripts (
    id TEXT PRIMARY KEY,
    text TEXT NOT NULL DEFAULT '',
    source TEXT NOT NULL DEFAULT 'microphone',
    title TEXT,
    audio_path TEXT,
    duration REAL NOT NULL DEFAULT 0.0,
    engine TEXT,
    model_id TEXT,
    inference_ms INTEGER,
    sample_rate INTEGER,
    audio_channels INTEGER,
    format_mode TEXT,
    remove_fillers INTEGER NOT NULL DEFAULT 0,
    british_english INTEGER NOT NULL DEFAULT 0,
    anti_hallucination INTEGER NOT NULL DEFAULT 0,
    created_at TEXT NOT NULL DEFAULT (datetime('now')),
    -- v5 (transcripts_meta):
    starred INTEGER NOT NULL DEFAULT 0,
    manual_tags TEXT NOT NULL DEFAULT '',
    template TEXT NOT NULL DEFAULT '',
    language TEXT NOT NULL DEFAULT '',
    segments_json TEXT NOT NULL DEFAULT '',
    -- v8/v9 (profile FK):
    profile_id TEXT NOT NULL REFERENCES profiles(id),
    -- v14:
    llm_tags TEXT NOT NULL DEFAULT ''
);

CREATE INDEX idx_transcripts_created ON transcripts(created_at);
CREATE INDEX idx_transcripts_profile_id ON transcripts(profile_id);
CREATE INDEX idx_transcripts_profile_created ON transcripts(profile_id, created_at DESC);  -- v15
```

### `segments` — keyed `INTEGER PRIMARY KEY AUTOINCREMENT`

```sql
CREATE TABLE segments (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    transcript_id TEXT NOT NULL REFERENCES transcripts(id) ON DELETE CASCADE,
    start_time REAL NOT NULL,
    end_time REAL NOT NULL,
    text TEXT NOT NULL DEFAULT ''
);
CREATE INDEX idx_segments_transcript ON segments(transcript_id);
```

(Note: today the engines store full segments in `transcripts.segments_json` rather than this table. The table is kept for backwards compatibility.)

### `transcripts_fts` — FTS5 virtual table (v2, rebuilt v9)

```sql
CREATE VIRTUAL TABLE transcripts_fts USING fts5(
    text, title,
    content='transcripts', content_rowid='rowid',
    tokenize='porter unicode61 remove_diacritics 2'
);
```

Plus three triggers (`transcripts_ai`, `transcripts_ad`, `transcripts_au`) that mirror inserts / deletes / updates from `transcripts` into the FTS index. See [`storage-fts5-search.md`](storage-fts5-search.md).

### `tasks`

```sql
CREATE TABLE tasks (
    id TEXT PRIMARY KEY,
    text TEXT NOT NULL,
    bucket TEXT NOT NULL DEFAULT 'inbox',
    list_id TEXT,
    effort TEXT,
    done INTEGER NOT NULL DEFAULT 0,
    done_at TEXT,
    created_at TEXT NOT NULL DEFAULT (datetime('now')),
    source_transcript_id TEXT,
    -- v3:
    parent_task_id TEXT REFERENCES tasks(id) ON DELETE CASCADE,
    -- v4:
    notes TEXT NOT NULL DEFAULT '',
    -- v11:
    energy TEXT,  -- CHECK (energy IN ('high', 'medium', 'brain_dead') OR energy IS NULL)
    -- v13:
    auto_completed INTEGER NOT NULL DEFAULT 0
);

CREATE INDEX idx_tasks_bucket ON tasks(bucket);
CREATE INDEX idx_tasks_transcript ON tasks(source_transcript_id);
CREATE INDEX idx_tasks_parent ON tasks(parent_task_id);
CREATE INDEX idx_tasks_energy_created ON tasks(energy, created_at);  -- v11
CREATE INDEX idx_tasks_done_at_auto_completed ON tasks(done_at, auto_completed);  -- v13
```

### `task_lists`

```sql
CREATE TABLE task_lists (
    id TEXT PRIMARY KEY,
    name TEXT NOT NULL,
    built_in INTEGER NOT NULL DEFAULT 0,
    profile_id TEXT,
    created_at TEXT NOT NULL DEFAULT (datetime('now'))
);
```

### `profiles` (v6)

```sql
CREATE TABLE profiles (
    id TEXT PRIMARY KEY,
    name TEXT NOT NULL UNIQUE,
    initial_prompt TEXT NOT NULL DEFAULT '',
    created_at TEXT NOT NULL DEFAULT (datetime('now'))
);
```

Default profile seeded at id `00000000-0000-0000-0000-000000000001`. Two SQL triggers protect it:

```sql
CREATE TRIGGER trg_protect_default_profile_delete
    BEFORE DELETE ON profiles
    WHEN OLD.id = '00000000-0000-0000-0000-000000000001'
    BEGIN SELECT RAISE(ABORT, 'cannot delete the default profile'); END;

CREATE TRIGGER trg_protect_default_profile_rename
    BEFORE UPDATE OF id, name ON profiles
    WHEN OLD.id = '00000000-0000-0000-0000-000000000001'
    BEGIN SELECT RAISE(ABORT, 'cannot rename the default profile'); END;
```

### `profile_terms` (v6)

```sql
CREATE TABLE profile_terms (
    id TEXT PRIMARY KEY,
    profile_id TEXT NOT NULL REFERENCES profiles(id) ON DELETE CASCADE,
    term TEXT NOT NULL,
    note TEXT,
    created_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE INDEX idx_profile_terms_profile_id ON profile_terms(profile_id);
```

UUIDs for `id` are generated as v4 random per `crates/storage/Cargo.toml`. (The `Cargo.toml` comment says "v7 random" but the feature flag is `["v4"]`. This is a known inconsistency between comment and reality. The code path uses `Uuid::new_v4()`.)

### `settings`

Plain key-value store. `key TEXT PRIMARY KEY, value TEXT NOT NULL`. The frontend's preferences blob lives at key `lumotia_preferences`.

### `error_log`

Append-only diagnostic log. Pruned by `prune_error_log` on app startup with a 90-day default. See [`storage-crud-settings-and-misc.md`](storage-crud-settings-and-misc.md).

### `feedback` (v10)

HITL thumbs + corrections, scoped to profile. See [`storage-crud-settings-and-misc.md`](storage-crud-settings-and-misc.md).

### `implementation_rules` (v12)

If-then automation rules. See [`storage-crud-settings-and-misc.md`](storage-crud-settings-and-misc.md).

### `schema_version`

Bookkeeping. Created by the migration runner itself.

## Migration history

15 migrations, all in `MIGRATIONS: &[(i64, &str, &str)]` at `crates/storage/src/migrations.rs:7`.

| v | Description | What changes |
|---|---|---|
| 1 | initial schema | Creates `transcripts`, `segments`, `tasks`, `task_lists`, `settings`, `error_log` plus 5 base indexes. |
| 2 | transcripts FTS5 + dictionary table | Creates `transcripts_fts` virtual table, three sync triggers, `dictionary` table (later removed). |
| 3 | micro-stepping: `parent_task_id` on tasks | `ALTER tasks ADD COLUMN parent_task_id` (with FK + cascade); `idx_tasks_parent`. |
| 4 | tasks_meta: notes column | `ALTER tasks ADD COLUMN notes TEXT NOT NULL DEFAULT ''`. |
| 5 | transcripts_meta | Adds `starred`, `manual_tags`, `template`, `language`, `segments_json` columns. Persists what was previously in `localStorage` `lumotia_history`. |
| 6 | profiles | Creates `profiles` and `profile_terms` tables; seeds the default profile; copies `dictionary` rows to the default profile's `profile_terms`; installs the two protection triggers. |
| 7 | drop_dictionary | Removes the legacy `dictionary` table and its index. Data has already been copied in v6. |
| 8 | transcript_profile_provenance | Adds `transcripts.profile_id` (nullable initially); `idx_transcripts_profile_id`. |
| 9 | transcript_profile_fk | **Table rebuild.** Reconciles orphaned `transcripts.profile_id` values to the default profile, then rebuilds `transcripts` and `segments` with the FK declared. The FTS table and triggers are dropped and recreated against the rebuilt `transcripts`. Driven by [`docs/issues/c4-transcript-profile-fk.md`](../../issues/c4-transcript-profile-fk.md). |
| 10 | feedback: HITL thumbs + correction capture | Creates `feedback` table; two indexes (target_type+rating, profile). |
| 11 | tasks: energy tagging | `ALTER tasks ADD COLUMN energy` with `CHECK` constraint; `idx_tasks_energy_created`. |
| 12 | implementation intentions | Creates `implementation_rules` table; `idx_implementation_rules_enabled_trigger`. |
| 13 | gamification: `auto_completed` | `ALTER tasks ADD COLUMN auto_completed`; `idx_tasks_done_at_auto_completed`. Cascade-completed parents are tagged so the daily-completion query can exclude them. |
| 14 | transcripts: `llm_tags` column | `ALTER transcripts ADD COLUMN llm_tags TEXT NOT NULL DEFAULT ''`. Phase 9 LLM content tags. |
| 15 | composite (profile_id, created_at DESC) index | `CREATE INDEX idx_transcripts_profile_created`. Speeds up the profile-scoped history page query that currently uses `idx_transcripts_profile_id` alone. |

## Tests

Migration tests live alongside the registry at `crates/storage/src/migrations.rs:606-1184`:

- `test_migrations_run_on_empty_db` — full v1→head replay.
- `test_migrations_idempotent` — running twice is a no-op.
- `migration_tasks_meta_adds_columns` — v3 ALTER goes through.
- `migration_implementation_rules_adds_rule_table` — v12 happy path.
- `migration_transcripts_meta_adds_columns` — v5 ALTER goes through.
- `migration_transcript_profile_provenance_adds_profile_id` — v8.
- `migration_v9_reconciles_orphaned_transcript_profiles_and_adds_fk` — the v9 table rebuild end-to-end.
- `test_parent_task_id_cascade_delete` — v3 cascade FK behaviour.
- `migration_v6_seeds_default_profile_on_fresh_db`.
- `migration_v6_copies_dictionary_rows_to_default_profile_terms`.
- `migration_v6_trigger_rejects_default_profile_delete`.
- `migration_v6_trigger_rejects_default_profile_rename`.
- `migration_v7_drops_dictionary_table`.
- `multi_statement_migration_rolls_back_on_failure` — the C3 atomicity guarantee.
- `migration_v13_adds_auto_completed_column`.
- `migration_v15_creates_profile_created_index` — the head migration.

## Watch-outs

- **Append-only.** Modifying an existing migration after release is a contract violation. Add a new one.
- **`split_statements` is hand-rolled.** Triggers contain `BEGIN ... END` with embedded semicolons. The splitter at `migrations.rs:483` is trigger-aware (it tracks `BEGIN` / `END` depth) but the parser is hand-rolled rather than a real SQL tokeniser. Adding a new control-flow keyword (eg `CASE WHEN ... END` outside a trigger) could trip it. Test additions are advised.
- **Migration v9 is the only table rebuild.** Rebuilds drop the FTS index and recreate it. If a future column needs FK adjustment, the v9 pattern is the template, but it is more risk than an `ALTER`.
- **No down migrations.** Recovery from a bad migration is restore-from-backup. The pre-commit pattern is to test on a fresh DB and against a copy of a real one.

## Existing in-repo docs

- [`docs/issues/c3-migrations-atomicity.md`](../../issues/c3-migrations-atomicity.md) — RB-02. Drove the per-migration transaction wrapper.
- [`docs/issues/c4-transcript-profile-fk.md`](../../issues/c4-transcript-profile-fk.md) — drove migration v9.

## See also

- [Storage overview](storage-overview.md)
- [FTS5 search](storage-fts5-search.md)
- [Transcripts CRUD](storage-crud-transcripts.md)
- [Tasks CRUD](storage-crud-tasks.md)
- [Profiles CRUD](storage-crud-profiles.md)
- [Settings, error log, feedback, implementation rules](storage-crud-settings-and-misc.md)