Log your data’s habits, not its secrets

What it does

whylogs generates compact statistical summaries—called profiles—from any dataset. These profiles capture distributions, missing values, and custom metrics without storing raw rows, letting you track data drift, validate pipeline constraints, and audit model inputs after the fact. It runs in both Python and Java, and the profiles are designed to merge so distributed or streaming workloads can be aggregated cleanly.

The interesting bit

Instead of sampling or shipping raw data, whylogs uses mergeable statistical sketches. You can generate profiles on separate workers or time slices, fuse them into a single aggregate, and still catch training-serving skew or concept drift without centralizing sensitive rows.

Key highlights

• Profiles capture distribution shapes, missing-value counts, and configurable custom metrics for structured or unstructured data
• Mergeable design supports distributed logging and arbitrary time-window aggregation
• Constraint rules can trigger failures in unit tests or CI/CD when data breaks expected ranges or formats
• Built-in Jupyter visualizer generates interactive drift reports and profile comparisons
• Optional WhyLabs SaaS integration turns profiles into automated monitoring alerts; a free starter tier exists

Caveats

• Java is supported, but every example and visualizer in the README is Python-first

Verdict

A solid fit for ML teams that need offline-capable, privacy-preserving data quality monitoring without building a full observability stack. Skip it if your platform already handles drift detection and constraint checking end-to-end.