Microsoft’s recipe for squeezing specialist performance from GPT-4

What it does promptbase is Microsoft’s research repository for advanced prompt engineering. It currently hosts reference implementations of the Medprompt and Medprompt+ methodologies, which combine dynamic few-shot retrieval, self-generated chain-of-thought reasoning, and choice-shuffle ensembling to squeeze stronger performance from foundation models on specialized benchmarks. The repo also includes evaluation scripts across standard tests like MMLU, GSM8K, and HumanEval, though these read more as methodological lab notes than production tools.

The interesting bit The project treats prompting as a systems composition problem rather than a lucky guess. Medprompt retrieves semantically similar training examples via k-NN in an embedding space, then auto-generates reasoning chains by asking GPT-4 to write its own steps. Medprompt+ adds a meta-cognitive filter: it asks the model whether a question requires a “scratch-pad” before dynamically weighting chain-of-thought against direct-answer prompts in the ensemble.

Key highlights

• Medprompt beat domain-specific fine-tuned models in medicine using only prompting strategies on a generalist GPT-4 model.
• Medprompt+ achieved a 90.10% score on the broad MMLU benchmark, marginally edging Gemini Ultra’s 90.04%.
• Uses dynamic few-shot selection via text-embedding-ada-002 k-NN clustering instead of static hand-crafted examples.
• Automates chain-of-thought generation by prompting GPT-4 to write its own reasoning steps, with guardrails against hallucinated logic.
• Employs choice-shuffle ensembling for multiple-choice questions, selecting answers least sensitive to option ordering.

Caveats

• The repository explicitly notes that some scripts are published for methodological reference and may not be immediately executable against public APIs.
• Medprompt+ relies on GPT-4 logprobs (confidence scores) that are not currently available via the public API, limiting out-of-the-box reproducibility.
• Running evaluations requires manually sourcing and formatting several external datasets.

Verdict Researchers and applied scientists who treat prompt engineering as an empirical optimization problem will find the methodology and ablation details valuable. If you are looking for a polished, drop-in SDK for your application, this is not it—yet.