Microsoft's LLM family that evolves its own training data

What it does

WizardLM is a collection of instruction-following models—generalist (WizardLM), coder (WizardCoder), and mathematician (WizardMath)—built by fine-tuning base models like Llama 2, Mistral 7B, and DeepSeek-Coder. The core trick is Evol-Instruct: using an LLM to iteratively rewrite existing instructions into more complex variants, then training on the harder set. The result is a zoo of checkpoints from 1B to 70B that benchmark well against ChatGPT 3.5 and Gemini Pro on coding and math tasks.

The interesting bit

The project treats data augmentation as an adversarial game against itself—an LLM mutates prompts to increase difficulty, then another LLM learns to solve them. It is also refreshingly candid about what it cannot share: the training data is stuck in legal review and not publicly available, despite the Apache 2.0 code license and open model weights.

Key highlights

• WizardCoder-33B-V1.1 hits 79.9% on HumanEval and outperforms ChatGPT 3.5 and Gemini Pro on several coding benchmarks, according to the project’s tables.
• WizardMath-7B-V1.1 claims SOTA among 7B models on GSM8k (83.2%) and beats ChatGPT 3.5 on the same benchmark.
• WizardLM and WizardCoder papers were accepted at ICLR 2024.
• Model weights are released on Hugging Face, but the underlying evolved datasets remain unreleased due to organizational policy changes.

Caveats

• Several smaller or older checkpoints carry non-commercial licenses, while newer ones use the Llama 2 License; check the specific model before shipping anything to production.
• The training data is explicitly not open-sourced—the authors state their “researchers have no authority to publicly release them without authorization.”
• Benchmark tables compare against specific dated versions of GPT-4 and ChatGPT; treat the “beats GPT-4” claims as snapshot-specific rather than categorical.

Verdict

Worth a look if you need open-weight coding or math models and can tolerate license heterogeneity. Skip it if you require reproducible training pipelines or fully open datasets—this is a model zoo, not a recipe book.