A feature store that automates the paranoid parts of ML pipelines

What it does

Feathr is a feature store that LinkedIn ran in production for over six years before open-sourcing it in 2022; it is now a sandbox project under the LF AI & Data Foundation. It lets teams define data transformations via Pythonic APIs against batch or streaming sources, registers those transformations by name, and automatically joins the resulting features to training data with point-in-time-correct semantics to prevent leakage. The same definitions can then be materialized for online serving, so research and production share a single source of truth.

The interesting bit

Rather than treating point-in-time correctness as a user-side burden, Feathr bakes it into the join engine, automatically handling window aggregations and sliding joins so data scientists don’t accidentally train on future information. It pairs this with a built-in registry and web UI for lineage tracking and access control, making feature reuse closer to governance than copy-paste.

Key highlights

• Proven at LinkedIn for more than six years on billions of rows and petabyte-scale workloads, using optimizations like bloom filters and salted joins.
• Native integrations with Azure Synapse and Databricks, plus a unified API that spans offline batch, streaming, and online serving.
• Custom UDFs with native PySpark and Spark SQL support, aimed at data scientists already working in the Spark ecosystem.
• Built-in registry and UI for searching features, inspecting lineage, and managing access controls.
• Derived features—such as embedding similarity—can be composed on top of existing features without redefining underlying transformations.

Caveats

• The project is a sandbox-tier member of the LF AI & Data Foundation, so governance and long-term roadmap stability are still maturing.
• Cloud deployment documentation centers heavily on Azure ARM templates and Azure CLI; Databricks is supported, but the narrative is Azure-first.
• The live UI demo requires a Microsoft work, school, or personal account, which is an unusual gate for an open-source tool.

Verdict

Data science teams running large-scale Spark workloads on Azure or Databricks who need point-in-time-correct feature joins and cross-team feature reuse should evaluate Feathr. If your pipeline is small-scale or lives outside the Spark and Azure orbit, this is probably not your tool.