Teaching machines to click the 'I am not a robot' boxes

What it does

Takes an image of a click-the-characters CAPTCHA—common on Chinese sites like Bilibili—and returns the exact coordinates to click, in the correct order. It handles both standard text-selection puzzles and “match-3” style variants. The system runs as a local REST API or Python module, and claims ~300–500 ms response times on CPU-only hardware as low as 1 core / 2 GB RAM.

The interesting bit

The two-stage architecture is what makes this work on modest hardware. First, a YOLO model finds all text regions and labels them as either background noise (char) or targets (target). Then a Siamese network compares each candidate region against a built-in character library to determine which glyphs match and what order to click them. Both models are converted to ONNX for CPU inference—no GPU required.

Key highlights

• Claims 96% accuracy and ~300 ms inference on CPU (per README; no independent benchmark cited)
• Ships with a Bilibili automation demo (bilbil.py) and a match-3 solver (xiaoxiaole.py)
• Training data requirement is unusually small: ~300 labeled images for custom CAPTCHA types
• Returns both bounding boxes and normalized click coordinates in JSON
• Includes a pre-packaged dataset via Baidu NetDisk (extraction code: sp97)

Caveats

• The “96% accuracy” and “300–500 ms” figures are self-reported with no test methodology or dataset details provided
• Heavily tied to a specific Chinese tutorial ecosystem (“道满PythonAI”); documentation and support channels are China-centric
• Disclaimer explicitly limits use to “academic research and learning exchange”—a reminder that CAPTCHA-solving sits in an ethical gray zone

Verdict

Worth a look if you’re building browser automation for Chinese web services or studying how lightweight CV pipelines can replace OCR-heavy approaches. Skip it if you need audited, production-grade accuracy claims or if your target sites use non-Chinese character sets—the built-in library and training data are Sino-centric.