A benchmark that tests fluid intelligence, not memorization

What it does

ARC-AGI-1 is a collection of 800 visual reasoning tasks—400 for training, 400 for evaluation—stored as plain JSON. Each task shows a few input-output pairs: colored grids where cells hold integers 0–9. The solver must infer the hidden transformation and reconstruct the exact output grid for a new test input, dimensions included. The repository also ships a browser-based interface (apps/testing_interface.html) that lets humans tackle the puzzles manually with paint, flood-fill, and resize tools.

The interesting bit

The format is intentionally austere—nothing but 30×30 grids of ten symbols—yet it is explicitly designed as a psychometric test for general fluid intelligence. By requiring solvers to deduce novel rules from a handful of examples, it draws a hard line between memorization and actual reasoning.

Key highlights

• 400 training and 400 evaluation tasks, served as self-contained JSON with no external knowledge required.
• Exact-match scoring: every cell and the output dimensions must be correct; partial credit is not an option.
• The browser testing interface supports copy-paste, floodfill, and grid resizing for quick human baseline testing.
• Grounded in the framework from On the Measure of Intelligence, which treats the corpus as both a program-synthesis benchmark and an IQ test for machines.

Caveats

• The manual interface does not enforce the official three-trials rule, so casual human scores may not match formal benchmark conditions.
• The repository is strictly task data plus a testing UI; it does not include reference implementations, training loops, or automated evaluation scripts.

Verdict

Ideal for researchers probing few-shot abstraction or anyone who wants a minimal, rigorous intelligence benchmark. Less useful if you are looking for a ready-made training framework or baseline model.