gkamradt/needle-in-a-haystack
Benchmarks LLMs on their ability to retrieve a hidden 'needle' fact from long 'haystack' contexts at varying positions.
Not currently ranked — collecting fresh signals.
star history
Needle-in-a-Haystack is an evaluation framework for pressure-testing LLM long-context retrieval. It runs controlled experiments sweeping context length against needle depth, scores each model response, and writes structured results to JSONL. Built-in tasks include single-fact lookup, multi-fact recall, and UUID-chain hops for multi-step reasoning. Supports OpenAI, Anthropic, and Cohere providers out of the box with a plugin system for adding more.
Frequently asked
- What is gkamradt/needle-in-a-haystack?
- Benchmarks LLMs on their ability to retrieve a hidden 'needle' fact from long 'haystack' contexts at varying positions.
- Is needle-in-a-haystack open source?
- Yes — gkamradt/needle-in-a-haystack is an open-source project tracked on heatdrop.
- What language is needle-in-a-haystack written in?
- gkamradt/needle-in-a-haystack is primarily written in Jupyter Notebook.
- How popular is needle-in-a-haystack?
- gkamradt/needle-in-a-haystack has 2.3k stars on GitHub.
- Where can I find needle-in-a-haystack?
- gkamradt/needle-in-a-haystack is on GitHub at https://github.com/gkamradt/needle-in-a-haystack.