A DataFrame Engine That Speaks Image, Audio, and LLM

What it does Daft is a query-optimized dataframe engine built in Rust with a Python face. It reads structured data alongside images, video, audio, and embeddings from sources like S3, Iceberg, Delta Lake, or Hugging Face, then runs vectorized operations including LLM prompting and embedding generation across local machines or distributed Ray and Kubernetes clusters.

The interesting bit Most dataframe libraries bolt multimodal data on as opaque Python objects; Daft’s own comparison table flags this as a weakness in Polars, Pandas, and Dask. By keeping the engine Arrow-backed and vectorized while natively understanding non-tabular types, it tries to bridge data engineering and AI inference without shuffling between separate systems.

Key highlights

• Native multimodal support: images, audio, video, and embeddings live alongside structured columns without being cast to generic Python objects
• Built-in AI operations: run LLM prompts, generate embeddings, and classify via OpenAI, Transformers, or custom models at scale
• Rust core, Python API: skips JVM complexity; execution engine is Rust under the hood with a Python-native surface
• Distributed by design: runs locally or scales to Ray and Kubernetes clusters with the same code
• Broad connectivity: S3, GCS, Iceberg, Delta Lake, Hugging Face, and Unity Catalog are supported out of the box
• Query optimizer and out-of-core execution: claims intelligent memory management and vectorized processing

Caveats

• The README mentions benchmarks but provides no specific numbers or reproduction details beyond a link to external docs
• Telemetry is collected by default via Scarf and must be explicitly disabled with DO_NOT_TRACK=true
• The “Related Projects” comparison table is authored by the Daft team, so take the feature matrix with appropriate skepticism

Verdict Teams building AI pipelines that churn through images, video, or embeddings alongside structured data should take a look. If your workloads are purely tabular and single-node, Polars or Pandas likely already cover you without the extra abstraction.