Git workflows for datasets that make git-lfs cry

What it does Oxen is a data version control system written in Rust that borrows git’s command vocabulary but reimplements the internals for scale. It targets repositories with millions of files and terabytes of unstructured or structured data, shipping as a CLI, a Rust crate, Python bindings, and an HTTP API backed by an optional server component.

The interesting bit Rather than treating everything as opaque blobs, Oxen runs specialized metadata extractors for formats like Parquet and caches the results inside a merkle tree for fast retrieval. It also supports remote workspaces that let you query and manipulate DataFrames without downloading the full dataset, and the server can emit Prometheus metrics and OpenTelemetry traces when compiled with the production feature flag.

Key highlights

• Claims to index hundreds of thousands of images in seconds
• Native DataFrame operations: index, compare, and serve tabular data directly from the repository
• Handles arbitrary blob types—images, video, audio, JSON, Arrow, models—with format-aware metadata extraction
• Remote workspaces allow interacting with data without a full local clone
• Production server builds add FFmpeg thumbnail generation, structured JSON log rotation, and OTLP-compatible tracing

Caveats

• macOS developers using mise for Python management may need to manually configure DYLD_LIBRARY_PATH to resolve dynamic library linking errors during builds
• Server-side features like Prometheus metrics, OpenTelemetry tracing, and FFmpeg thumbnails require compiling with the production feature flag; they are excluded from the default development build to keep binaries smaller

Verdict ML teams whose datasets have outgrown git-lfs should look here; if your data fits comfortably in a standard git repo, you likely don’t need another VCS.