Data diagnosis that skips the training bill

What it does

Docta is a Python toolkit that inspects datasets for “unhealthy” entries—specifically mislabeled examples and underrepresented patterns—across tabular, image, and text data. It generates diagnosis reports by analyzing features extracted from pre-trained models, and the open-source release includes four training-free demos. One demo audits Anthropic’s HH-RLHF alignment data, flagging thousands of potentially incorrect harmlessness ratings.

The interesting bit

Instead of training a custom model to find errors, Docta diagnoses embeddings you already have (or extract on the fly). In the HH-RLHF demo, it suggests roughly 8% of red-team attempts and 28% of harmless-base pairs carry human annotation errors—including a Molotov-cocktail recipe that workers rated low-harm but the algorithm flags as severe.

Key highlights

• Supports tabular, image, text, and pre-trained model embeddings
• Claims to operate without any additional model training on your data
• Includes ready-made demos for CIFAR-N, CIFAR-10, tabular data, and Anthropic HH-RLHF
• Flags both label errors and low-frequency “rare patterns” the authors compare to overlooked vitamins
• Non-commercial license (CC BY-NC 4.0)

Caveats

• Licensed under Creative Commons Attribution-NonCommercial 4.0; commercial use requires contacting the authors
• The authors explicitly warn that sampled results are algorithm-generated and should be used “at your own risk”
• Feature extraction strongly recommends a GPU, and the tool expects local data preparation and config tweaking

Verdict

Worth a look if you’re curating RLHF or image-classification datasets and want a quick sanity check on labels without spinning up a training run. Skip it if you need a commercial-grade license or a fully hands-off tool—this still expects local data prep and human review of its flagged results.