What it does

LLaMA Factory is a unified fine-tuning framework for large language and vision-language models. It wraps pre-training, supervised fine-tuning, reward modeling, PPO, DPO, KTO, and ORPO behind a single CLI or Gradio web interface called LLaMA Board. It also spits out an OpenAI-style API for inference via vLLM or SGLang.

The interesting bit

The breadth is almost comical: 2-bit through 16-bit quantization via AQLM, AWQ, GPTQ, HQQ, and others; LoRA variants you’ve never heard of (DoRA, LongLoRA, PiSSA, LoftQ); and day-zero support for new models like Qwen3 and Llama 4. It’s less a research tool than a logistics platform for keeping up with model releases.

Key highlights

• Supports 100+ models including LLaMA, Qwen, DeepSeek, Gemma, GLM, and multimodal variants like LLaVA and Qwen-VL
• Integrates GaLore, BAdam, APOLLO, Adam-mini, Muon, and other recent optimizers
• Includes FlashAttention-2, Unsloth, and Liger Kernel for speed
• Docker images, Colab notebooks, and cloud integrations (SageMaker, PAI-DSW) ready to go
• Used by Amazon, NVIDIA, and Aliyun according to their own case studies

Caveats

• Documentation is marked “WIP” and the README warns about unauthorized third-party sites
• The sheer surface area means you’re trusting one codebase to correctly implement dozens of algorithms

Verdict

Grab this if you need to fine-tune frequently across different model families and don’t want to maintain separate pipelines. Skip it if you’re doing novel research that needs surgical control over training dynamics—this is integration, not invention.