rlancemartin/auto-evaluator
An evaluation tool for measuring the quality of LLM-based question-answering chains built with Langchain.
Not currently ranked — collecting fresh signals.
star history
Auto-evaluator generates question-answer pairs from documents using LLMs, runs them through configurable QA chains built with Langchain, and uses LLMs to score the responses. Users can experiment with text splitting, embedding methods, retrieval strategies, and grading prompts to compare chain performance across configurations. It runs as a Streamlit app and supports models from OpenAI, Anthropic, and Hugging Face.
Frequently asked
- What is rlancemartin/auto-evaluator?
- An evaluation tool for measuring the quality of LLM-based question-answering chains built with Langchain.
- Is auto-evaluator open source?
- Yes — rlancemartin/auto-evaluator is an open-source project tracked on heatdrop.
- What language is auto-evaluator written in?
- rlancemartin/auto-evaluator is primarily written in Python.
- How popular is auto-evaluator?
- rlancemartin/auto-evaluator has 1.1k stars on GitHub.
- Where can I find auto-evaluator?
- rlancemartin/auto-evaluator is on GitHub at https://github.com/rlancemartin/auto-evaluator.