An agent factory that iterates on its own products

What it does

SIA is an orchestration framework that runs a self-improvement loop over AI agents. A meta-agent spawns a task-specific target agent; after each run, a feedback agent reviews the logs and rewrites the target’s code and weights for the next generation. The loop repeats until the benchmark score plateaus or you hit your generation cap. It ships with four built-in tasks—gpqa, lawbench, longcot-chess, and spaceship-titanic—and accepts custom task directories or MLE-Bench competitions.

The interesting bit

The twist is that the improvement agent edits both the harness and the weights themselves, so the system is not just prompting harder—it is restructuring how the agent works. The README cites paper results like a 56.6% gain on LawBench and a 14× CUDA kernel speedup, though your mileage will depend heavily on the base models and evaluators you plug in.

Key highlights

• Three-agent architecture: a meta-agent drafts the worker, a target agent executes the task, and a feedback agent proposes edits for the next generation.
• Built-in web dashboard that visualizes per-generation code diffs, accuracy charts, and execution trajectories live.
• Pluggable profiles and providers let you swap models and agent implementations without touching core code.
• Automatic evaluation after each generation feeds scores directly into the next feedback prompt.
• Bundled support for MLE-Bench competitions and a template for bringing entirely custom tasks.

Caveats

• The README is truncated mid-document, cutting off the evaluation guide and leaving some referenced docs unseen.
• Reported benchmark numbers come from the paper; how easily they translate to your own tasks and model choices is unclear.
• The system relies on external LLM APIs for its meta and feedback agents, so running many generations can rack up tokens quickly.

Verdict

Worth a look if you’re running repetitive agent benchmarks and want to automate the prompt-engineering-and-refactoring cycle. Skip it if you need a single-shot, deterministic solver or if your task lacks a clear automated evaluator, since the entire loop depends on a numeric feedback signal.