openai/human-eval
Benchmark harness for evaluating the functional correctness of code completions generated by large language models.
Not currently ranked — collecting fresh signals.
star history
This repository contains the evaluation harness for the HumanEval dataset, a benchmark for assessing code-generating language models. It provides tooling to run model-generated Python code in a sandboxed environment and measure functional correctness against hand-written test cases. The dataset and methodology were introduced in the paper ‘Evaluating Large Language Models Trained on Code’ and is widely used to compare code-assistance models like Codex.
Frequently asked
- What is openai/human-eval?
- Benchmark harness for evaluating the functional correctness of code completions generated by large language models.
- Is human-eval open source?
- Yes — openai/human-eval is open source, released under the MIT license.
- What language is human-eval written in?
- openai/human-eval is primarily written in Python.
- How popular is human-eval?
- openai/human-eval has 3.3k stars on GitHub.
- Where can I find human-eval?
- openai/human-eval is on GitHub at https://github.com/openai/human-eval.