LLM Grid Search Without the Python Boilerplate

What it does

Ollama Grid Search is a desktop application that treats your local or remote Ollama server like a test lab. You define a matrix of models, prompts, and inference parameters; the app runs every combination and presents the results in a visual grid for side-by-side inspection. It also archives past experiments so you can rerun or tweak them later without reconstructing the setup from memory.

The interesting bit

The project borrows the term “grid search” from ML training, but applies it to inference-time decisions—model selection, prompt phrasing, and inference parameters. That reframing turns a training optimization concept into a practical tool for prompt engineering and model evaluation. The Rust-plus-React stack also keeps the whole thing self-contained as a native desktop app rather than yet another browser-based notebook workflow.

Key highlights

• Automatically discovers models from local or remote Ollama endpoints.
• Runs A/B tests across multiple models or prompt variations in one shot.
• Supports limited concurrency or synchronous calls to avoid spamming servers.
• Logs every experiment as downloadable JSON, with options to rerun or clone previous parameter sets.
• Includes a prompt archive with / autocomplete, inspired by Open WebUI.

Caveats

• The macOS build uses ad-hoc code signing, so Apple Silicon users need to right-click and manually approve the first launch; it is not notarized.
• The README admits that “grid search” is technically a misnomer borrowed from training hyperparameter tuning, which may confuse purists.
• Future features like grading results and importing/exporting prompt lists are listed but not yet implemented.

Verdict

Anyone running a local Ollama instance and tired of manual model comparisons should grab this. If you are looking for a server-side evaluation pipeline or cloud-native benchmarking at scale, this is not your tool.