hegelai/prompttools
Open-source Python library for experimenting with, testing, and evaluating prompts, LLMs, and vector databases via code, notebooks, and a local playground.

The repository provides developer tooling for running prompt experiments across multiple LLM providers and vector databases. It exposes Python APIs like OpenAIChatExperiment that let developers programmatically test prompts against different models, compare outputs, and evaluate retrieval accuracy. The tool supports major providers including OpenAI, Anthropic, and LLaMA, and integrates with vector databases such as Chroma, Weaviate, and LanceDB for semantic search evaluation.