carlini/yet-another-applied-llm-benchmark
A practical benchmark for evaluating language models on real-world tasks derived from the author's own LLM usage.
Not currently ranked — collecting fresh signals.
star history
A benchmark framework that tests LLMs on diverse applied tasks like code generation, parsing, decompilation, and format identification. It uses a custom dataflow DSL where tests chain operations via the » operator, combining LLM execution, code running (in Docker containers), and output evaluation. Includes nearly 100 test cases covering tasks the author has actually asked LLMs to perform.
Frequently asked
- What is carlini/yet-another-applied-llm-benchmark?
- A practical benchmark for evaluating language models on real-world tasks derived from the author's own LLM usage.
- Is yet-another-applied-llm-benchmark open source?
- Yes — carlini/yet-another-applied-llm-benchmark is open source, released under the GPL-3.0 license.
- What language is yet-another-applied-llm-benchmark written in?
- carlini/yet-another-applied-llm-benchmark is primarily written in Python.
- How popular is yet-another-applied-llm-benchmark?
- carlini/yet-another-applied-llm-benchmark has 1.1k stars on GitHub.
- Where can I find yet-another-applied-llm-benchmark?
- carlini/yet-another-applied-llm-benchmark is on GitHub at https://github.com/carlini/yet-another-applied-llm-benchmark.