THUDM/LongBench
A benchmark suite for evaluating LLMs on challenging long-context tasks requiring deep understanding and reasoning across single/multi-document QA, in-context learning, dialogue understanding, and code comprehension.
LongBench v2 provides a comprehensive evaluation framework for assessing large language models on realistic long-context multitasks. It contains 503 multiple-choice questions with context lengths ranging from 8k to 2M words, covering six major task categories including single-document QA, multi-document QA, long in-context learning, long-dialogue history understanding, code repo understanding, and long structured data understanding. The benchmark is designed to be challenging enough that even human experts using search tools cannot answer quickly, ensuring meaningful evaluation of deep reasoning capabilities.
Frequently asked
- What is THUDM/LongBench?
- A benchmark suite for evaluating LLMs on challenging long-context tasks requiring deep understanding and reasoning across single/multi-document QA, in-context learning, dialogue understanding, and code comprehension.
- Is LongBench open source?
- Yes — THUDM/LongBench is open source, released under the MIT license.
- What language is LongBench written in?
- THUDM/LongBench is primarily written in Python.
- How popular is LongBench?
- THUDM/LongBench has 1.2k stars on GitHub.
- Where can I find LongBench?
- THUDM/LongBench is on GitHub at https://github.com/THUDM/LongBench.