Train a 64M-parameter LLM from scratch in two hours and three yuan

What it does

MiniMind is a fully open-source reproduction of a 64-million-parameter language model small enough to train on a single NVIDIA 3090. It provides the complete stack: tokenizer training, architecture code for Dense and MoE variants (aligned with the Qwen3 ecosystem), pre-training, supervised fine-tuning, LoRA, DPO, and RL algorithms including PPO, GRPO, and CISPO. The repository also includes curated datasets, evaluation scripts for benchmarks like C-Eval and OpenBookQA, and an OpenAI-compatible API server with a Streamlit chat interface.

The interesting bit

The author implements every core algorithm from scratch in native PyTorch without relying on transformers, trl, or peft abstractions for the internals, yet the resulting checkpoints still plug into those frameworks plus llama.cpp, vllm, and ollama. This creates a rare educational hybrid: it is explicitly a tutorial for understanding LLM mechanics, but the outputs are real enough to serve through standard tooling. The README itself clarifies that the famous “two hours and three yuan” claim applies specifically to one SFT epoch, not the entire pipeline from random weights.

Key highlights

• Architecture variants include a 64M Dense model and a 198M-A64M MoE model, plus experimental offshoots for vision, diffusion language modeling, and linear attention.
• Training covers the full modern lifecycle: pre-training, SFT, tool-use integration, agentic RL with multi-turn tool calls, adaptive thinking chains, and model distillation.
• Core math—LoRA, DPO, PPO, GRPO, and YaRN long-context extrapolation—is handwritten in PyTorch rather than imported from high-level libraries.
• Supports distributed training via DDP and DeepSpeed, with visualization hooks for wandb and SwanLab.
• Recent releases have broken backward compatibility; pre-2025 checkpoints are no longer directly loadable and the v1 model family is unmaintained.

Caveats

• The “two hours and three yuan” benchmark refers specifically to the SFT stage running one epoch on a single NVIDIA 3090, not the full pre-training and RL pipeline.
• Older model checkpoints released before April 2025 are no longer directly loadable due to architectural refactoring, and the v1 series has been abandoned.
• The project is explicitly educational; a 64M-parameter model is useful for learning internals, not for replacing production LLMs.

Verdict

Grab this if you are a developer who wants to stop treating LLMs as black boxes and actually touch the matrix math. Skip it if you need a drop-in foundation model competitive with modern production LLMs.