Peer review in a petri dish, bias dials included
You can't run controlled experiments on real peer review — the data is private and every variable is confounded — so this EMNLP 2024 oral paper builds a tunable LLM-agent stand-in instead.

What it does AgentReview stages a full conference review cycle with LLM agents in every seat: three reviewers, an author, and an area chair, moving through five phases — independent reviews, rebuttals, reviewer–AC discussion, meta-review, and the final accept/reject call. The substrate is real: 523 ICLR 2020–2023 submissions pulled via the OpenReview API, with the acceptance rate pinned at 32% to match the conference’s actual average. Reviewer commitment, intention, knowledgeability, author anonymity, and AC decision style are all prompt-level dials, and each experiment turns exactly one of them against a baseline.
The interesting bit The framework is essentially an ablation machine for a social process: flip one reviewer from responsible to irresponsible and watch altruism fatigue drag the whole panel’s commitment down 18.7%. The headline result is that 37.1% of decision variation is attributable to reviewer bias alone. And the runs leave behind a corpus — 53,800+ generated review documents, including 10,460 reviews and rebuttals.
Key highlights
- Quantifies five sociological effects: social influence (−27.2% rating spread after rebuttals), altruism fatigue (one slacking reviewer → −18.7% commitment panel-wide), groupthink spillover (−0.25 rating even on unbiased reviewers), authority bias, and anchoring.
- Revealing author identities for just 10% of papers shifts 27.7% of decisions — anonymity is doing quiet, heavy lifting.
- Rebuttals barely move final outcomes; the initial review anchors the decision. Condolences to everyone who has written a rebuttal at 3 a.m.
- New experiments are entries in
agentreview/experiment_config.py, not new code — it builds on the chatarena multi-agent framework. - A live Gradio demo on Hugging Face means you can poke it before committing to anything.
Caveats
- Needs an OpenAI or Azure OpenAI key, and the README warns that provider content filtering can mangle generations — you may have to relax it on your deployment.
- The numbers describe LLM reviewers, not human ones; they’re theory-grounded hypotheses about real review, not measurements of it.
- This is paper infrastructure, notebook-heavy, with the demo living in
notebooks/demo.ipynb— built for research, not as a product.
Verdict Care if you study or reform peer review, or want a ready-made harness for controlled multi-agent experiments. Look elsewhere if you wanted a robot to review your paper — this is an instrument for studying the process, not a participant in it.
Frequently asked
- What is Ahren09/AgentReview?
- You can't run controlled experiments on real peer review — the data is private and every variable is confounded — so this EMNLP 2024 oral paper builds a tunable LLM-agent stand-in instead.
- Is AgentReview open source?
- Yes — Ahren09/AgentReview is open source, released under the Apache-2.0 license.
- What language is AgentReview written in?
- Ahren09/AgentReview is primarily written in Jupyter Notebook.
- How popular is AgentReview?
- Ahren09/AgentReview has 719 stars on GitHub.
- Where can I find AgentReview?
- Ahren09/AgentReview is on GitHub at https://github.com/Ahren09/AgentReview.