Meta's Columnar Format for the Thousand-Column Table

What it does Nimble stores large columnar datasets in a binary layout optimized for tables that sprawl across thousands of columns—think feature-engineering and ML training workloads. It replaces Parquet and ORC with a C++ library that uses Flatbuffers for metadata and block encoding instead of stream encoding to keep memory usage predictable while reading. Meta positions it as a product, not merely a specification, and actively discourages third-party reimplementations to avoid the fragmentation that plagued earlier formats.

The interesting bit The format decouples stream encoding from the physical file layout, so users can plug in custom encodings and even stack them recursively—cascading encodings—without waiting for a spec revision. That architecture treats encoding as an extensible plugin system rather than a frozen format decision.

Key highlights

• Targets “wide” workloads with thousands to tens of thousands of columns and streams.
• Uses Flatbuffers for metadata access, claiming efficiency over Thrift or Protobuf for large metadata sections.
• Block-based encoding aims for predictable memory usage during decoding, contrasting with stream-based approaches.
• Supports pluggable and recursively composable (cascading) encodings.
• Currently tightly coupled with the Velox query engine, though the team plans to decouple them.

Caveats

• Still a work in progress with no stability or versioning guarantees yet.
• SIMD and GPU-friendly encodings are on the roadmap but not implemented yet.
• The codebase is currently closely coupled with Velox, which may complicate adoption outside Velox-centric stacks until a future decoupling.

Verdict Data infrastructure teams running wide ML feature stores should keep an eye on this; if you need a drop-in Parquet replacement today, this is not it.