Skip to content

corral

A generic engine for tabular data packages in the Frictionless format. Lazy by default, regional-scale ready, with composable primitives for validation, scope, editing, and serving.

What problems corral solves

Reading any Frictionless data package, regardless of physical format, with one API surface:

from corral import Package

pkg = Package.from_source("local/path/datapackage.json")      # local directory
pkg = Package.from_source("s3://bucket/path/datapackage.json")  # cloud, credentials cascade
pkg = Package.from_source("./mydata.duckdb")                    # single-file duckdb
pkg = Package.from_source("./mydata.csv.zip")                   # zipped CSV bundle

Lazy evaluation at regional scale. Tables aren’t materialised until you ask. pkg.tables["link"].filter(...).count() pushes the count to DuckDB; only the integer comes back to Python.

Engine choice without API rewrite. The same Package.from_source(...).validate() works against ibis (default), pandas, or polars. Switch via engine= per call.

Validation as a single report. pkg.validate() returns one ValidationReport covering structural, schema, foreign-key, and sync-state checks. Severity-graded; rendered as rich console, JSON, or interactive HTML.

Composable editing with rollback. Open a Session, apply one or many edits, commit or roll back atomically. Persisted log as a sidecar parquet file.

Generic surfaces — CLI, FastAPI HTTP server, and MCP server — all reusable for any domain-specific spec that builds on corral (netstead being the canonical example).

Use cases — when to install dbcorral directly

  •  GTFS interop research

    Building a GTFS ↔ GMNS bridge, or analysing GTFS feeds with the same toolchain you use for other tabular specs.

  •  Custom internal spec

    Your organisation has a tabular data spec (sensor metadata, asset catalogs, planning datasets) that you want to validate, version, and serve consistently.

  •  Cloud-resident data

    Tables live in S3 / Azure Blob / GCS, and you want lazy SQL pushdown without writing the connection plumbing yourself.

  •  Building your own toolkit

    You want the generic primitives (engine ABC, FormatAdapter registry, ValidationReport, Session, FastAPI helpers) to compose into a domain-specific package. netstead is the worked example.

Why install dbcorral

It’s small and focused. Generic data-package primitives only — no domain semantics. Easy to reason about, easy to extend.

Backend-agnostic. ibis for SQL pushdown by default, pandas for compatibility, polars for fast in-memory. Add a backend by implementing the Engine protocol; no API rewrite.

Production-grade defaults. Bearer-token auth on the HTTP server by default. Warn-loudly on misconfiguration (e.g. auth=none + non-localhost bind). Cost-model gating on long operations with explicit approval semantics.

Extension points are first-class. register_adapter for new formats. register_engine for new backends. register_rule for quality rules. extra_router_factory for HTTP extensions. Same pattern across the surface.

Install

Pick the tool you already use — these all produce the same install:

uv add dbcorral

Fastest. Works inside a uv-managed project and writes to your pyproject.toml + uv.lock.

uv pip install dbcorral

Drop-in pip replacement. Use this in a plain venv without a project file.

pip install dbcorral

Classic. Works anywhere Python does.

pipx install dbcorral

Isolated env for the corral CLI only — your project env stays untouched.

Optional extras

The default install ships the ibis + DuckDB engine and Frictionless loader. Extras let you opt in to specific engines, cloud backends, and the AI surface:

Extra Pulls in When you need it
polars polars>=1.0 Use the polars engine for in-memory speed (see engines decision guide)
pandas pandas>=2.2 Use the pandas engine for DataFrame ergonomics
s3 / gcs / azure corresponding fsspec adapter Read from cloud-storage URLs
keyring keyring>=24 Resolve credentials from the system keychain
mcp mcp>=1.0 Run corral mcp serve for Claude Desktop / Code

Install with the same syntax (uv shown — substitute your tool):

uv add 'dbcorral[polars,s3,keyring]'   # combine with commas

zsh users: quote the brackets

On zsh (the default shell on macOS), [ and ] are glob characters. Running uv add dbcorral[polars] unquoted gives zsh: no matches found: corral[polars]. Always wrap the extras in quotes ('corral[polars]' or "corral[polars]"), or run setopt no_nomatch once per session to disable the check. bash users don’t hit this.

Where to go next

  •  Quickstart

    Load a data package, validate it, scope it, write it out — in five minutes.

    Get started

  •  Cookbook

    Task-oriented recipes — read from S3, convert formats, spatial scope.

    Browse recipes

  •  API reference

    Every public symbol, auto-generated from docstrings, with stable anchors.

    API reference

  • :material-architecture:{ .lg .middle }  Architecture

    Defaults, rationales, extension points. Single source of truth.

    Architecture