← all repositories
Ahren09/AgentReview

Peer review in a petri dish, bias dials included

You can't run controlled experiments on real peer review — the data is private and every variable is confounded — so this EMNLP 2024 oral paper builds a tunable LLM-agent stand-in instead.

★719 stars Jupyter Notebook AgentsDomain AppsLanguage Models
AgentReview
Not currently ranked.
star history

What it does AgentReview stages a full conference review cycle with LLM agents in every seat: three reviewers, an author, and an area chair, moving through five phases — independent reviews, rebuttals, reviewer–AC discussion, meta-review, and the final accept/reject call. The substrate is real: 523 ICLR 2020–2023 submissions pulled via the OpenReview API, with the acceptance rate pinned at 32% to match the conference’s actual average. Reviewer commitment, intention, knowledgeability, author anonymity, and AC decision style are all prompt-level dials, and each experiment turns exactly one of them against a baseline.

The interesting bit The framework is essentially an ablation machine for a social process: flip one reviewer from responsible to irresponsible and watch altruism fatigue drag the whole panel’s commitment down 18.7%. The headline result is that 37.1% of decision variation is attributable to reviewer bias alone. And the runs leave behind a corpus — 53,800+ generated review documents, including 10,460 reviews and rebuttals.

Key highlights

  • Quantifies five sociological effects: social influence (−27.2% rating spread after rebuttals), altruism fatigue (one slacking reviewer → −18.7% commitment panel-wide), groupthink spillover (−0.25 rating even on unbiased reviewers), authority bias, and anchoring.
  • Revealing author identities for just 10% of papers shifts 27.7% of decisions — anonymity is doing quiet, heavy lifting.
  • Rebuttals barely move final outcomes; the initial review anchors the decision. Condolences to everyone who has written a rebuttal at 3 a.m.
  • New experiments are entries in agentreview/experiment_config.py, not new code — it builds on the chatarena multi-agent framework.
  • A live Gradio demo on Hugging Face means you can poke it before committing to anything.

Caveats

  • Needs an OpenAI or Azure OpenAI key, and the README warns that provider content filtering can mangle generations — you may have to relax it on your deployment.
  • The numbers describe LLM reviewers, not human ones; they’re theory-grounded hypotheses about real review, not measurements of it.
  • This is paper infrastructure, notebook-heavy, with the demo living in notebooks/demo.ipynb — built for research, not as a product.

Verdict Care if you study or reform peer review, or want a ready-made harness for controlled multi-agent experiments. Look elsewhere if you wanted a robot to review your paper — this is an instrument for studying the process, not a participant in it.

Frequently asked

What is Ahren09/AgentReview?
You can't run controlled experiments on real peer review — the data is private and every variable is confounded — so this EMNLP 2024 oral paper builds a tunable LLM-agent stand-in instead.
Is AgentReview open source?
Yes — Ahren09/AgentReview is open source, released under the Apache-2.0 license.
What language is AgentReview written in?
Ahren09/AgentReview is primarily written in Jupyter Notebook.
How popular is AgentReview?
Ahren09/AgentReview has 719 stars on GitHub.
Where can I find AgentReview?
Ahren09/AgentReview is on GitHub at https://github.com/Ahren09/AgentReview.

heatdrop uses Google Analytics to see which pages get read — nothing else. Your call. How we handle data.