Databox

Databox is a local-first ingestion and data-product platform with dlt-managed Iceberg raw tables on AWS S3, a local Apache Polaris catalog, and DuckDB transformations. SQLMesh transforms data, Soda validates it, and Dagster orchestrates the workflow.

Start here

Evaluate offline

task install   # creates .env from .env.example when absent
task ci

This runs the repository's static, test, secret, and generated-artifact gates. It does not require a provider refresh or populated warehouse.

Build and inspect the warehouse

Configure source, Polaris, and AWS S3 writer values in the .env created by task install, then run:

$EDITOR .env
docker compose --env-file .env -f compose.iceberg.yml up -d
task full-refresh   # ingest Iceberg raw tables → local models → quality
task dagster:dev    # inspect assets at localhost:3000

Raw tables live in S3 and are registered by local Polaris; modeled data and export inputs live in data/databox.duckdb. See configuration, the commands reference, and the operations runbook.

Understand the data

• Data dictionary — models, columns, types, contracts, and direct lineage
• Lineage — the complete model dependency graph
• Analytics examples — representative CDM queries
• Metrics — canonical semantic metrics
• Contracts — Soda quality conventions

Add a source

Start with the new-source workflow, then follow the authoritative source layout. After ingestion, the project skills move each source through annotation and taxonomy, ontology, Kimball CDM design, and SQLMesh transformation. The registry-derived modeling guard verifies that every source completes this chain.

Operating and extending

• Incremental loading
• Environments
• Freshness
• Observability
• CI routing
• Protected Polaris/S3 verification — manual real-provider integration gate
• Architecture decisions

Everything under dictionary/ is generated from SQLMesh and Soda metadata. Do not hand-edit it.