A file format that finally speaks fluent AI

What it does

Lance is a lakehouse storage format built for multimodal AI workloads. It stores tables on object storage with native support for images, video, audio, text, and embeddings alongside structured data. The project provides a file format, table format, and catalog spec with Python, Java, and Rust bindings.

The interesting bit

The format inverts the usual priority stack. Traditional lakehouse formats (Parquet, Iceberg, Delta Lake) optimize for analytical scans; Lance optimizes for the messy reality of ML pipelines where you need random access, vector search, and schema evolution in the same breath. It claims 100× faster random access than Parquet without sacrificing scan performance, and bundles ACID versioning, hybrid search, and lazy blob loading into the format itself rather than bolting them on via external systems.

Key highlights

• Hybrid search combining vector similarity, BM25 full-text, and SQL predicates with secondary indices
• Batched random access benchmarked at 100× faster than Parquet on the Oxford Pet dataset
• Sub-millisecond vector search latency on SIFT-1M (128D vectors) per project benchmarks
• Schema evolution with column addition and backfill without full table rewrites
• Zero-copy versioning with time travel, tags, and branches—no extra infrastructure
• Broad integration surface: Arrow, Pandas, Polars, DuckDB, Spark, Ray, Trino, Flink, plus open catalogs

Caveats

• Benchmarks are self-reported and hardware-specific (2023 M2 MacBook Air for vector search)
• The README notes DuckDB integration can segfault on older versions, suggesting binding roughness
• “Active development” disclaimer implies API stability is not guaranteed

Verdict

Worth evaluating if you’re building feature stores, search engines, or training pipelines that juggle embeddings with unstructured data. Stick with Parquet or Iceberg if your workload is purely SQL analytics and you don’t need vector search or fast random sampling.