Open-source models beat GPT-4 Omni when forced to collaborate

What it does

Mixture-of-Agents (MoA) is a multi-tier inference pipeline that passes prompts through successive layers of open-source LLMs. Each layer feeds its collective outputs into an aggregator model, which refines the response before passing it upward. The result is a composite answer that, according to the project’s benchmarks, outperforms single closed-source models on instruction-following tasks.

The interesting bit

The architecture borrows from neural network design but swaps neurons for entire language models: early layers generate diverse candidate responses, and later layers act as hidden units that synthesize those candidates into a final output. The paper demonstrates that this stack of relatively small open models can score 65.1% on AlpacaEval 2.0, a 7.6 point absolute margin over GPT-4 Omni.

Key highlights

• Achieved 65.1% on AlpacaEval 2.0 using only open-source weights, beating GPT-4 Omni’s 57.5%
• FLASK fine-grained evaluation shows MoA topping Qwen1.5-110B-Chat on harmlessness, robustness, correctness, efficiency, factuality, commonsense, insightfulness, and completeness
• Also exceeds GPT-4 Omni on FLASK dimensions including correctness, factuality, insightfulness, completeness, and metacognition
• Ships with reproducible evaluation scripts for AlpacaEval, MT-Bench, and FLASK, though these bundle modified third-party code
• Includes a minimal ~50-line reference implementation (moa.py), a multi-layer variant (advanced-moa.py), and an interactive chat demo (bot.py)

Caveats

• Every example and demo is wired to the Together AI API; the README documents no alternative backends or local execution path
• Evaluation reproduction relies on modified copies of AlpacaEval, MT-Bench, and FLASK bundled in subdirectories, so staying current with upstream fixes is left as an exercise
• The README omits any discussion of per-query latency or token costs despite a design that calls multiple large models across several layers

Verdict

Benchmark hunters and open-source researchers who want to composite models into a stronger whole should take a close look. If you need predictable unit economics or low-latency responses, the README offers little guidance on whether this architecture fits.