The curated index to the code LLM benchmark wars

What it does This repository is a curated index of the best large language models for code generation, aimed at researchers who want to compare models without digging through a dozen papers and Hugging Face pages. It ranks models by HumanEval and MBPP scores in a single table, and links out to evaluation frameworks, active leaderboards, and research papers on pre-training, instruction-tuning, and alignment. Think of it as a living bibliography with a scoreboard attached.

The interesting bit The README doesn’t just list models; it sorts them by HumanEval Pass@1, which makes the gap between closed-weight giants like o1-mini and open-source challengers like Qwen2.5-Coder-32B brutally clear. It also surfaces niche evaluation resources—like SandboxFusion for secure code execution and the Aider code-editing leaderboard—that go beyond standard benchmark hype.

Key highlights

• Ranked table of 25+ models with HumanEval and MBPP scores, from o1-mini down to Code-Davinci-002.
• Links to evaluation toolkits including bigcode-evaluation-harness, code-eval, and SandboxFusion.
• A sprawling leaderboard directory covering efficiency (Evalperf), real-world editing (Aider), SQL (BIRD), and contamination-free testing (LiveCodeBench).
• Curated paper lists broken down by pre-training, instruction-tuning, alignment, prompting, and benchmark design.
• Recent news updates tracking releases like the Qwen2.5-Coder series and the OpenCoder cookbook.

Caveats

• The list is explicitly a curation, not a benchmark itself; all scores are pulled from external papers and repositories.
• Some leaderboard descriptions are thin—CanAiCode Leaderboard is literally described as “CanAiCode Leaderboard”—so you may still need to click through.

Verdict Researchers and engineers choosing a code model for a project should bookmark this; if you are looking for a tutorial or an API quickstart, this is just a well-organized index.