Teaching LLaMA to chat by letting GPT interview itself

What it does

Baize fine-tunes LLaMA into an open-source chatbot using LoRA adapters. The repository holds training code, data-collection scripts that seed synthetic conversations from Quora, Stack Overflow, and MedQuAD, and released weights for 7B, 13B, and 30B parameter models. A Gradio demo and FastChat integration provide interfaces for testing.

The interesting bit

The core trick is data bootstrapping: the authors generated roughly 100k dialogs by prompting ChatGPT to interview itself, then blended those with Alpaca data to teach LLaMA conversational turns. The full pipeline runs on a single A100-80G GPU, with the 7B model finishing in about nine hours and the 30B variant in roughly thirty-six.

Key highlights

• Publishes both LoRA weight deltas (v1) and merged standalone checkpoints (v2), including a healthcare-specific 7B model
• Training VRAM requirements stay manageable via 8-bit quantization: 26 GB for 7B up to 42 GB for 30B
• Inference starts at 16 GB VRAM for the 7B model without quantization
• Community derivatives include official Falcon instruct models trained on Baize data, plus Italian and Dutch forks
• Licensed under GPL-3.0 with an explicit research-only restriction on weights and data

Caveats

• Weights and data carry a strict research-only, non-commercial clause with no liability accepted—unusual baggage for an open release
• v2 model code was listed as “coming soon” as of May 2023, so the repo may trail the Hugging Face checkpoints
• The included demo is a fork of ChuanhuChatGPT, meaning the UI is borrowed scaffolding rather than original work

Verdict

A solid reference if you want to distill ChatGPT-style conversation into LLaMA using nothing more than a single GPU. Avoid if you need commercially usable weights or a turnkey product instead of a training recipe.