← all repositories
molly554/whiteboard-ai-tutor

The AI tutor that can't draw straight — and the benchmark proving it

A whiteboard tutor protocol plus a 288-run blind-reviewed benchmark showing that LLMs get the words right and the pictures wrong.

whiteboard-ai-tutor
Too new to rank. Velocity needs a few days of history.
not enough history yet
star history

What it does

The repo is two things: a reference implementation of a whiteboard tutoring loop, and a benchmark of six models doing it. The model never touches a canvas — it streams JSON drawing operations (write, line, rect, ellipse, path, erase) on a 1600×900 board, and a browser player renders them as handwriting. When a student interrupts, the app sends the board state (every object’s id, position, text) back so the model can erase or annotate by id. The benchmark side ran one frozen prompt across eight scenarios, six models, two thinking settings, three repeats — 288 lessons, 64 boards blind-reviewed, total cost $4.83.

The interesting bit

The headline finding is that the words are almost never wrong; the pictures are. Of 20 failed boards, 12 failed because the figure contradicted the narration — a season label on the wrong Earth, an equilibrium point sitting beside the intersection instead of on it. That’s a failure mode you won’t see in any text-only eval, and it’s the reason this benchmark exists.

Key highlights

  • Gemini 4 Argon passed 11/11 blind reviews; Claude Haiku 5.5 passed 9/11 at 8–15× lower cost; Claude Sonnet 5.5 drew first (2.8 s to first stroke) with the cleanest layout
  • DeepSeek V4.1 Flash (thinking off) returned JSON missing the op field in 7 of 24 calls — parses fine, draws nothing
  • The automatic linter turned out to be a bad judge: it flagged 41 boards as “hard errors” that blind review then passed 26 of
  • Turning thinking on mostly bought waiting — 30–120 s to first stroke, and 23 calls hit the 8,000-token output cap mid-board
  • Interruptions worked but messily: all 24 boards answered the student, yet 11 wrote over the existing drawing or wiped the board
  • Every reviewed board is replayable in a live gallery, and the player is a single HTML file with no build step

Caveats

  • Eleven boards per model is a small sample — the top-to-bottom gap is clear, the middle ordering is not (the README says so itself)
  • The blind reviewer was Claude Fable 5.2, an LLM checking geometry from pixel positions; a human audit of 18 boards exists but is a sample, not a full re-review
  • It’s not a hosted tutoring app — the player replays recorded lessons; generating new ones requires running the benchmark runner with your own API keys

Verdict

Worth a look if you build LLM-driven teaching tools or care about evaluating multimodal output beyond text. If you wanted a ready-to-use tutor product, this isn’t it — it’s the protocol and the evidence, and the evidence is the fun part.

Frequently asked

What is molly554/whiteboard-ai-tutor?
A whiteboard tutor protocol plus a 288-run blind-reviewed benchmark showing that LLMs get the words right and the pictures wrong.
Is whiteboard-ai-tutor open source?
Yes — molly554/whiteboard-ai-tutor is open source, released under the MIT license.
What language is whiteboard-ai-tutor written in?
molly554/whiteboard-ai-tutor is primarily written in HTML.
How popular is whiteboard-ai-tutor?
molly554/whiteboard-ai-tutor has 546 stars on GitHub.
Where can I find whiteboard-ai-tutor?
molly554/whiteboard-ai-tutor is on GitHub at https://github.com/molly554/whiteboard-ai-tutor.

heatdrop uses Google Analytics to see which pages get read — nothing else. Your call. How we handle data.