An AI text detector that admits it's not very good

What it does

This repo contains two things: a lightweight, offline CLI/Python analyzer (ai-detect) that scores text 0–100 for AI-likeness using weighted, explainable signals, and a production two-level model pipeline (document + sentence classification with XLM-R or ModernBERT backbones) that ships as code, not weights. The local analyzer is packaged as a skill for Codex and Claude Code, so agents can run it inside repo workflows without touching the network.

The interesting bit

The README publishes its own unflattering HC3 evaluation: mean separation of 13 points between human and AI text, covered accuracy of 0.317. That honesty is the feature — the tool is built to say “this is a weak signal, review it yourself” rather than “this was definitely AI,” and the short-text guardrail refuses to score samples under ~80 words.

Key highlights

• Runs fully offline; no network calls, no model hosting for the local analyzer
• Output includes signals, caveats, and next_steps — evidence, not just a number
• Verdicts are deliberately hedged: mixed_or_uncertain is a first-class outcome
• Production pipeline is a full train → calibrate → serve flow with temperature scaling and confidence bands
• Reproducible evaluation on the public HC3 dataset, regenerated in CI

Caveats

• The local analyzer is weak as a classifier — the authors themselves frame it as triage, not detection
• The production model ships as pipeline code only; no weights or datasets are included, so you’d have to train it yourself or buy the hosted API
• The README doubles as a sales funnel for the Lynote API, which is worth knowing going in

Verdict

Worth a look if you want an honest, scriptable triage signal for editorial or moderation queues — or a reference for how to document a detector’s limitations. Skip it if you need actual accuracy; the numbers say you won’t get it here.