Teaching LLMs to grade their own homework

What it does

Self-Refine is a research artifact that wires large language models into a closed feedback loop: the model writes something, critiques it with a separate prompt, then revises based on that self-generated feedback. The repo implements this pattern across a handful of benchmark tasks—acronym generation, code readability, math word problems (GSM-8k), sentiment transfer, and even TikZ diagram generation—each driven by triads of Init, Feedback, and Iterate prompts.

The interesting bit

The project treats self-criticism not as a single correction but as a repeatable subroutine, effectively turning a monolithic generator into a tiny actor-critic system using nothing but prompt engineering. The recent addition of visual self-refine with GPT-4V shows the same loop can critique and redraw diagrams like Stokes’ theorem illustrations.

Key highlights

• Pure prompting strategy: no fine-tuning, just three prompt templates per task (task_init, feedback, task_iterate).
• Covers diverse domains: from Python readability and program improvement (PIE) to constrained generation (Commongen) and dialogue.
• Visual iteration: extends the loop to GPT-4V-generated TikZ code, refining diagrams like unicorn sketches.
• Built on prompt-lib: acts as glue code orchestrating calls to OpenAI models rather than a standalone framework.

Caveats

• The codebase is research glue code, not a reusable library; expect manual environment tweaks and task-specific data prep.
• It depends on an external prompt-lib repository, so it functions more as a collection of experiment scripts than a standalone package.
• The README never explains the stopping criteria; the mermaid diagram labels a decision diamond, but the actual logic remains unstated.

Verdict

Grab this if you’re researching iterative reasoning or building agentic loops with off-the-shelf LLMs. Skip it if you need a pip-installable package with clean APIs for production use.