bigcode-project/bigcodebench
A benchmark for evaluating large language models on code generation tasks with a public leaderboard.
Not currently ranked — collecting fresh signals.
star history
BigCodeBench is a code generation benchmark published at ICLR'25 that evaluates how well LLMs perform at synthesizing programs with tool use and function calling. It provides a standardized evaluation harness, a public leaderboard on HuggingFace, Docker-based evaluation infrastructure, and benchmarks LLMs across multiple dimensions including instruction-following, code completion, and agentic tool-use scenarios.
Frequently asked
- What is bigcode-project/bigcodebench?
- A benchmark for evaluating large language models on code generation tasks with a public leaderboard.
- Is bigcodebench open source?
- Yes — bigcode-project/bigcodebench is open source, released under the Apache-2.0 license.
- What language is bigcodebench written in?
- bigcode-project/bigcodebench is primarily written in Python.
- How popular is bigcodebench?
- bigcode-project/bigcodebench has 517 stars on GitHub.
- Where can I find bigcodebench?
- bigcode-project/bigcodebench is on GitHub at https://github.com/bigcode-project/bigcodebench.