Turning ML papers into code repos without the all-nighter

What it does PaperCoder ingests a machine learning paper—via LaTeX or a PDF converted to JSON—and spins up a trio of specialized LLM agents to plan, analyze, and generate a full code repository. The system targets the gap between published research and runnable implementations, producing structured output rather than a single script.

The interesting bit Instead of one monolithic model prompt, the work is split across dedicated agents for planning, analysis, and coding, which theoretically keeps architecture decisions separate from low-level tensor wrangling. The authors also provide a model-based evaluator that scores generated repos on a 1–5 scale, offering both reference-free and reference-based critique modes.

Key highlights

• Multi-agent pipeline with distinct planning, analysis, and code-generation agents
• Works with OpenAI’s o3-mini or open-source models via vLLM (defaulting to deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct)
• Accepts LaTeX directly or PDFs routed through an external JSON converter
• Includes a benchmark dataset and an automated evaluator that averages 1–5 correctness scores over 8 samples
• Example run costs roughly $0.50–$0.70 using OpenAI’s o3-mini

Caveats

• Running on your own paper requires modifying environment variables; the included demo targets the Transformer paper
• PDF inputs depend on an external s2orc-doc2json toolchain rather than native parsing
• The automated evaluator itself requires an OpenAI API key, so quality assessment is not fully self-contained

Verdict Researchers who need quick baselines from recent papers or want to stress-test LLM agent coordination will find this useful. If you already have the author’s official repo bookmarked, it’s probably overkill.