docs: architecture map (initial 5-slice generation, 105 pages)
Five-slice navigable map of the entire codebase under
docs/architecture-map/. Each slice is a self-contained
breadcrumbed sub-tree:
01-frontend (16) Svelte/SvelteKit UI
02-tauri-runtime (26) src-tauri commands + lifecycle
03-audio-transcription (16) audio + transcription crates
04-llm-formatting-mcp (19) llm, ai-formatting, mcp, cloud
05-core-storage-hotkey-build core, storage, hotkey, workspace,
(26) CI, dev glue
Plus master README.md and data-flow-end-to-end.md tracing
audio bytes from microphone to FTS5 search to MCP read.
Generated by 5 parallel subagents on 2026/05/09 against
HEAD 3c47000. Each page has YAML frontmatter, file:line code
refs, sibling cross-links, plain-English summaries.
Aggregated debt surfaced (full lists in master README):
RB-08 macOS power assertion, schema head drift v14 vs v15,
VAD blocked on ort version conflict, streaming primitives
not wired into live.rs, no prompt versioning, MCP has no
auth, cloud-providers in-memory keystore, SettingsPage
2 484 LOC, commands/live.rs 1 737 LOC, dual theme system,
brand rename to Lumenote pending across the codebase.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jars
@@ -0,0 +1,98 @@
|
||||
---
|
||||
name: Storage overview
|
||||
type: architecture-map-page
|
||||
slice: 05-core-storage-hotkey-build
|
||||
last_verified: 2026/05/09
|
||||
---
|
||||
|
||||
# Storage overview
|
||||
|
||||
> **Where you are:** [Architecture map](../README.md) → [Core, Storage, Hotkey, Build](README.md) → Storage overview
|
||||
|
||||
**Plain English summary.** The storage crate owns Magnotia's SQLite database, its FTS5 transcript index, the file-system paths it writes to, and the migration machinery that brings a fresh DB up to head. It uses `sqlx 0.8` in a slimmed-down configuration that strips out features the crate does not need.
|
||||
|
||||
## At a glance
|
||||
|
||||
- Crate: `magnotia-storage`.
|
||||
- LOC: 3,771 (database 2,534, migrations 1,185, file_storage 28, lib 24).
|
||||
- External deps: `sqlx 0.8` (`runtime-tokio`, `sqlite`; **no default features**), `tokio 1`, `serde 1`, `log 0.4`, `uuid 1` (v4), `magnotia-core` (path).
|
||||
- Public surface: 46 `pub async fn` (every CRUD verb listed in `crates/storage/src/lib.rs`), one `pub fn` (`as_str`), 9 `pub struct`s (param types + row types), 1 `pub enum` (`FeedbackTargetType`), 1 `pub const` (`DEFAULT_PROFILE_ID`), plus the file-storage path re-exports.
|
||||
- Consumers: every Tauri command that persists or reads (slice 2); the MCP server (slice 4) opens the same database read-only via `init_readonly`.
|
||||
|
||||
## sqlx configuration
|
||||
|
||||
```toml
|
||||
sqlx = { version = "0.8", default-features = false, features = ["runtime-tokio", "sqlite"] }
|
||||
```
|
||||
|
||||
`default-features = false` strips sqlx's `any`, `macros`, `migrate`, and `json` features. None of these are used:
|
||||
|
||||
- `any` — query builder for "any database backend"; we are SQLite-only.
|
||||
- `macros` — compile-time-checked queries; we use runtime queries via `sqlx::query()` and `sqlx::query_scalar()`.
|
||||
- `migrate` — sqlx's bundled migration runner; we run our own custom migration machinery (see [`storage-schema-and-migrations.md`](storage-schema-and-migrations.md)).
|
||||
- `json` — JSON column adapters; we serialise JSON into TEXT columns via `serde_json::to_string` at the call site.
|
||||
|
||||
This cuts ~40% of sqlx's compile graph, most visibly on Windows MSVC where every proc-macro crate (which `macros` pulls in) becomes a slow .dll link. Documented inline at `crates/storage/Cargo.toml`.
|
||||
|
||||
## Init flow
|
||||
|
||||
### `init(db_path)` — `crates/storage/src/database.rs:9`
|
||||
|
||||
The standard read-write entry point. Steps:
|
||||
|
||||
1. `std::fs::create_dir_all(parent)` so a fresh install has the parent directory.
|
||||
2. `SqliteConnectOptions::new().filename(db_path).create_if_missing(true)`.
|
||||
3. `SqlitePoolOptions::new().max_connections(5).connect_with(options)`.
|
||||
4. `PRAGMA foreign_keys = ON` — without this, sqlite ignores `REFERENCES` clauses at runtime even though they parse.
|
||||
5. `run_migrations(&pool)` — applies any pending migrations. See [`storage-schema-and-migrations.md`](storage-schema-and-migrations.md).
|
||||
|
||||
Returns a `SqlitePool` that is held in `tauri::State` for the rest of the process.
|
||||
|
||||
### `init_readonly(db_path)` — `crates/storage/src/database.rs:40`
|
||||
|
||||
The MCP server's entry point. Same options minus `create_if_missing(true)` and plus `read_only(true)`. `max_connections = 2` because the MCP server is single-purpose.
|
||||
|
||||
The structural read-only constraint matters: it makes "the MCP server cannot mutate user data" a property of the connection rather than a property of the dispatcher being well-behaved. Even if a future MCP tool tries to `INSERT`, sqlite refuses at the connection level.
|
||||
|
||||
## Public surface (lib.rs re-exports)
|
||||
|
||||
The complete public CRUD surface is re-exported from `crates/storage/src/lib.rs`:
|
||||
|
||||
- **Transcripts:** `insert_transcript`, `get_transcript`, `list_transcripts`, `list_transcripts_paged`, `count_transcripts`, `update_transcript`, `update_transcript_meta`, `delete_transcript`, `search_transcripts`.
|
||||
- **Tasks:** `insert_task`, `list_tasks`, `get_task_by_id`, `update_task`, `set_task_energy`, `complete_task`, `uncomplete_task`, `delete_task`. Plus subtasks: `insert_subtask`, `list_subtasks`, `complete_subtask_and_check_parent`. Plus completion analytics: `list_recent_completions`.
|
||||
- **Profiles:** `list_profiles`, `get_profile`, `create_profile`, `update_profile`, `delete_profile`. Plus profile terms: `list_profile_terms`, `add_profile_term`, `delete_profile_term`.
|
||||
- **Settings:** `set_setting`, `get_setting`.
|
||||
- **Error log:** `log_error`, `prune_error_log`, `list_recent_errors`.
|
||||
- **Feedback (HITL):** `record_feedback`, `list_feedback_examples`.
|
||||
- **Implementation rules:** `insert_implementation_rule`, `list_implementation_rules`, `get_implementation_rule`, `set_implementation_rule_enabled`, `mark_implementation_rule_fired`, `delete_implementation_rule`.
|
||||
- **Init:** `init`, `init_readonly`.
|
||||
- **File paths (re-exported from `file_storage.rs`):** `app_data_dir`, `database_path`, `recordings_dir`, `crashes_dir`, `logs_dir`.
|
||||
- **Constants:** `DEFAULT_PROFILE_ID = "00000000-0000-0000-0000-000000000001"`.
|
||||
|
||||
Per-table CRUD is split across the per-page docs in this slice. See:
|
||||
|
||||
- [Transcripts CRUD](storage-crud-transcripts.md)
|
||||
- [Tasks CRUD](storage-crud-tasks.md)
|
||||
- [Profiles CRUD](storage-crud-profiles.md)
|
||||
- [Settings, error log, feedback, implementation rules](storage-crud-settings-and-misc.md)
|
||||
- [FTS5 search](storage-fts5-search.md)
|
||||
- [Schema and migrations](storage-schema-and-migrations.md)
|
||||
- [File paths](storage-file-paths.md)
|
||||
|
||||
## Watch-outs
|
||||
|
||||
- **`PRAGMA foreign_keys = ON` is per-connection, not per-database.** The pool's `max_connections = 5` means we run the pragma once at init on the first connection. SQLite re-applies the pragma on each new pool connection because we set it via the connect options... but actually we don't, we set it after `connect_with`. **This is a latent issue worth verifying:** if a second pool connection opens later, foreign keys may not be enforced on it. Audit candidate.
|
||||
- **No connection-level retry on locked DB.** `SQLITE_BUSY` propagates as `MagnotiaError::StorageError(...)`. With WAL mode + 5 max connections this is rare, but a long-running write under a slow filesystem could trigger it.
|
||||
- **Custom migration runner.** sqlx's bundled `migrate!` macro is not used. The custom runner is documented in [`storage-schema-and-migrations.md`](storage-schema-and-migrations.md) and was the subject of the C3 critical-issue write-up at `docs/issues/c3-migrations-atomicity.md`.
|
||||
|
||||
## Existing in-repo docs
|
||||
|
||||
- [`docs/issues/c3-migrations-atomicity.md`](../../issues/c3-migrations-atomicity.md) — drove the transactional migration design.
|
||||
- [`docs/issues/c4-transcript-profile-fk.md`](../../issues/c4-transcript-profile-fk.md) — drove migration v9.
|
||||
|
||||
## See also
|
||||
|
||||
- [Schema and migrations](storage-schema-and-migrations.md)
|
||||
- [File storage paths](storage-file-paths.md)
|
||||
- [Slice 2 Tauri startup](../02-tauri-runtime/README.md) — the caller of `init`.
|
||||
- [Slice 4 MCP server](../04-llm-formatting-mcp/README.md) — the caller of `init_readonly`.
|
||||
Reference in New Issue
Block a user