arthur-ai/bench
A tool for evaluating and benchmarking LLMs against production use cases.
Not currently ranked — collecting fresh signals.
star history
Bench is a Python library for evaluating LLMs in production contexts. It provides standardized interfaces for running test suites against different LLMs, comparing prompt variations, and measuring generation hyperparameters like temperature and token count. Users create test suites with reference outputs and evaluate candidate model responses against them, enabling side-by-side comparison of open-source versus closed-source LLM performance.
Frequently asked
- What is arthur-ai/bench?
- A tool for evaluating and benchmarking LLMs against production use cases.
- Is bench open source?
- Yes — arthur-ai/bench is open source, released under the MIT license.
- What language is bench written in?
- arthur-ai/bench is primarily written in TypeScript.
- How popular is bench?
- arthur-ai/bench has 428 stars on GitHub.
- Where can I find bench?
- arthur-ai/bench is on GitHub at https://github.com/arthur-ai/bench.