Automated Prompt Tuning, Rated Like Chess

What it does

gpt-prompt-engineer is a set of Jupyter notebooks that automate the tedious cycle of writing and testing system prompts. You describe a task and provide test cases; the notebooks use an LLM to spawn dozens of prompt candidates, run them through your tests, and score the results. It is essentially orchestration code wrapped around OpenAI and Anthropic APIs.

The interesting bit

Instead of arbitrary scores, the tool ranks prompts with an ELO rating system—each candidate starts at 1200 and gains or loses points based on head-to-head performance against your test cases, turning prompt selection into a tournament. A newer variant also uses Claude 3 Opus to generate high-quality few-shot examples that steer Claude 3 Haiku, aiming to preserve output quality while reducing latency and cost.

Key highlights

• Generates prompt candidates using GPT-4, GPT-3.5-Turbo, or Claude 3 Opus.
• Ranks results with an ELO system rather than arbitrary scores.
• Includes a dedicated classification notebook (currently binary-focused based on the provided examples).
• Optional logging to Weights & Biases and Portkey for tracing chains.
• The Claude 3 version auto-generates test cases and supports multiple input variables.

Caveats

• It is notebook-based glue code, not an installable library, so wiring it into a production pipeline will take some elbow grease.
• The README explicitly warns that generating many prompts can burn through API credits quickly.
• Several links in the README (license, project URL) are still boilerplate templates.

Verdict

Worth a look if you are stuck hand-tuning prompts for a specific task and want data-driven rankings. Skip it if you need a polished SDK or are allergic to Google Colab.