The context layer that makes blob storage agent-ready

What it does DataChain is a Python library that indexes files in S3, GCS, or Azure into local, typed datasets backed by Pydantic schemas. It stores metadata and file pointers in a local SQLite database—called the Dataset DB—so you can filter, join, and run similarity search over millions of records without downloading data. Every transformation produces a named, versioned dataset with automatic lineage, and the optional Knowledge Base surfaces these as markdown summaries readable by both humans and LLMs.

The interesting bit The project treats your storage bucket as an immutable source of truth and builds a local “data harness” around it—analogous to how code harnesses give AI agents repo context. Agents like Claude Code or Cursor can decompose tasks into chain operations, save intermediate results as versioned datasets, and resume from previous runs instead of recomputing embeddings or metadata from scratch. Even crashes are handled gracefully: the compute engine checkpoints batch state, so a failed UDF restarts from the last uncommitted chunk rather than the beginning.

Key highlights

• Queries run vectorized against a local SQLite store, so filtering or joining across hundreds of millions of file records stays sub-second and never pulls bytes from cloud storage.
• Pydantic models define dataset schemas, including nested structures like bounding boxes inside annotations; every leaf becomes a queryable column enforced at creation time.
• Incremental updates via delta=True skip unchanged files on re-runs, while automatic versioning means pipelines reference datasets by name rather than brittle file paths.
• An optional Agent Harness plugs into Claude Code, Cursor, Codex, GitHub Copilot, and Pi via an MCP server, giving those agents persistent memory of your data context across sessions.
• Checkpoint recovery tracks processing state in the Dataset DB, letting buggy scripts resume from the last good batch without manual intervention.

Verdict DataChain is worth a look if you are building data pipelines or agent workflows over large, unstructured cloud storage and are tired of rewriting the same ingestion boilerplate. If your data already lives in a proper warehouse or you only ever touch small local files, the extra abstraction layer will likely feel like overkill.