C++ DataFrame loads 2B rows where Polars taps out at 300M

What it does DataFrame is a C++23 library for in-memory tabular data exploration, transformation, and statistical analysis. It treats columns as first-class citizens in contiguous memory, exposing joins, merges, group-bys, filters, and multi-column sorts. The author claims its breadth of functionality exceeds that of Pandas, R data.frame, and Polars combined.

The interesting bit Instead of wrapping Python or relying on external dependencies, the library is self-contained down to the C++ standard library. Algorithms arrive as composable “visitors” — from moving averages to FFT, PCA, and trading indicators — and columns can nest other DataFrames or containers, so the structure is not strictly two-dimensional.

Key highlights

• Zero external dependencies; the library relies solely on C++23 and the standard library.
• Heterogeneous columns accept any built-in or user-defined type without extra boilerplate.
• Built-in analytical visitors include PCA, polynomial fitting, FFT, eigenvalue decomposition, and trading indicators.
• Multithreading is woven through most APIs, but only for large datasets where overhead pays off.
• A synthetic benchmark on 300 million rows showed calculation times of 1.26 seconds, compared with 4.88 for Polars and 40.33 for Pandas; the author also reports loading 2-billion-row datasets that exhausted Polars’ memory.

Caveats

• The performance comparison is a single synthetic test (mean, variance, correlation, and one select) run on one MacBook Pro; it is not a comprehensive suite.
• The library explicitly follows a “garbage in, garbage out” philosophy, meaning little hand-holding for invalid inputs.
• The API is broad enough to warrant a cheat sheet and extensive HTML documentation, so expect a learning curve.

Verdict C++ quant developers and engineers wrestling with multi-gigabyte in-memory datasets should evaluate this. If your workflow is already fluent in Python and your data fits comfortably in Polars, the switching cost is likely too high.