Chat data brewed from Turbo APIs, not the web

What it does

UltraChat is a corpus of roughly 1.57 million multi-round dialogues generated with Turbo APIs, organized into three sectors: questions about the world, writing and creation, and assistance with existing materials like rewriting or summarization. The team uses this synthetic data to train UltraLM, a family of chat models built on LLaMA-13B and LLaMA-65B.

The interesting bit

Rather than scraping forums or social media, the dataset is synthesized from scratch to avoid privacy leaks and benchmark contamination. The resulting UltraLM-13B briefly ranked first among open-source models on AlpacaEval, and the later UltraRM reward model pushes best-of-16 sampling to a 92.30% win rate against text-davinci-003.

Key highlights

• 1.57M multi-round dialogues across three curated sectors
• Generated via Turbo APIs without using internet data as prompts
• UltraLM-13B and UltraLM-65B chat models trained on the corpus
• Ranked #1 among open-source models on AlpacaEval (June 2023)
• Ecosystem expanded to include reward model UltraRM and critic model UltraCM

Caveats

• The authors warn that scaling synthetic generation may still cause benchmark contamination despite avoiding public datasets.
• Reconstructing UltraLM requires original LLaMA weights plus separately released delta weights; third-party LLaMA redistributions are explicitly untested.

Verdict

Ideal for researchers who want a large, permissive chat dataset and the weights to reproduce a competitive LLaMA-based chat model. Less useful if you need a fully packaged, ready-to-run model without wrangling base-model licensing.