Skip to main content

Storage and Data

This page covers how Caracal stores operational state and how that state is resolved at runtime.

Audience: Developer

Context: Use this page when changing workspace behavior, configuration loading, database models, schema management, caching, or integrity pipelines.

Core Explanation​

Storage layout​

caracal/storage/layout.py defines the canonical CARACAL_HOME layout. Key directories include:

  • keystore/
  • workspaces/
  • ledger/
  • ledger/merkle/
  • ledger/audit_logs/
  • system/metadata/
  • system/history/

Permissions are enforced as part of layout creation.

Workspace manager​

caracal/flow/workspace.py resolves the active workspace and exposes paths for:

  • config.yaml
  • flow_state.json
  • backups/
  • logs/
  • cache/
  • keys/

It also maintains a registry of workspaces and default workspace selection.

Configuration loading​

caracal/config/settings.py builds CaracalConfig from defaults plus YAML plus environment variables plus optional encrypted values.

Important behavior:

  • missing config falls back to generated defaults
  • environment variables expand recursively inside config values
  • encrypted values are decrypted before validation
  • workspace-local backup and log paths are enforced
  • Merkle signing keys are auto-created when absent

PostgreSQL Schema​

caracal/db/models.py defines the core schema:

  • principals
  • execution_mandates
  • delegation_edges
  • authority_policies
  • authority_ledger_events
  • ledger_events
  • audit_logs
  • merkle_roots
  • ledger_snapshots
  • sync-related tables
  • provider records for gateway-facing integrations

caracal/db/connection.py treats PostgreSQL as mandatory and can optionally route all tables through a workspace schema.

Redis And Cache Paths​

caracal/redis/client.py provides the runtime Redis abstraction.

RedisMandateCache caches serialized mandates with TTL derived from the mandate's remaining validity. That makes cache expiry align with the authority window itself.

Merkle Integrity Data​

caracal/merkle layers integrity features on top of persisted events:

  • tree construction
  • batch accumulation
  • root signing
  • proof verification
  • snapshot and recovery helpers
  • backfill workflows

This is why the storage story is not just workspace files plus PostgreSQL. Integrity data is part of the runtime model.

Migrations And Schema Management​

The repository uses Alembic plus helper utilities under caracal/db and caracal/deployment.

Important paths:

  • caracal/db/migrations/
  • alembic.ini
  • caracal/db/schema_version.py
  • caracal/deployment/migration.py

Workspace deletion and import/export paths can also trigger PostgreSQL schema drop or restore operations.

Internal Behavior​

Several storage behaviors are intentionally strict:

  • PostgreSQL connection failure is a hard failure
  • schema names are validated
  • destructive operations are explicit methods, not implicit cleanup
  • workspace export can include a PostgreSQL schema dump when the workspace config defines one
  • locked workspace archives derive an AES key from a user-provided lock key

Edge Cases And Constraints​

  • Some legacy compatibility fields remain in config structures, but Redis and Merkle are not optional in the supported model.
  • WorkspaceManager.db_path exists only as a legacy compatibility property and does not represent a supported SQLite backend.
  • Import and restore workflows may require PostgreSQL client tools such as pg_dump, pg_restore, and psql.
AI tools
On this page