An LLM That Proposes, Runs, and Writes Up Experiments

What it does The AI Scientist is an autonomous pipeline that attempts end-to-end research within fixed templates. Starting from a provided codebase—such as a transformer or diffusion model—it brainstorms modifications, implements them, executes experiments, and produces a typeset PDF with citations and figures. It also includes an LLM-based peer review module to critique its own output.

The interesting bit The system closes the loop between hypothesis and validation by actually running the code it writes to gather results, then formatting everything into a paper. The authors shared dozens of generated papers publicly, offering a frank look at what fully automatic production currently looks like.

Key highlights

• Operates across three maintained domains: transformer language modeling (nanoGPT), low-dimensional diffusion, and grokking studies.
• Supports frontier closed and open-weight models, including GPT-4o, Claude 3.5 Sonnet, DeepSeek, and Gemini.
• Performs its own literature search via Semantic Scholar or OpenAlex to find real citations.
• Includes an automated peer-review stage where an LLM evaluates the generated paper.
• The authors ran approximately 50 ideas per template and published the resulting papers.

Caveats

• It executes LLM-generated code unsupervised, which carries explicit warnings about dangerous packages, web access, and process spawning; containerization is strongly advised.
• Requires Linux, NVIDIA GPUs, and CUDA; CPU-only execution is described as infeasible.
• Only the three core templates are maintained by the authors; any others are unsupported community contributions.

Verdict Worth a look if you study automated science or want a grounded demonstration of autonomous LLM agents. Skip it if you need a general-purpose researcher that works outside narrow ML benchmarks.