The anti-black-box: coding a GPT from the ground up

What it does This is the official code companion to Sebastian Raschka’s book Build a Large Language Model (From Scratch). It walks through constructing a small-but-functional GPT-like model using only PyTorch—no external LLM libraries—starting from text tokenization and culminating in instruction finetuning. The code is designed to run on conventional laptops, though it will use a GPU if one happens to be available.

The interesting bit The value is in the tedium: every attention head, layer norm, and training loop is written out explicitly in Jupyter notebooks so you can see the tensors mutate step by step. It is essentially a deliberate exercise in not using transformers or similar convenience wrappers.

Key highlights

• Implements the full stack: tokenization, causal multi-head attention, GPT architecture, pretraining, classification finetuning, and instruction tuning.
• Includes appendices on PyTorch basics, distributed data parallel, and parameter-efficient finetuning with LoRA.
• Ships with exercise solutions and a 17-hour companion video course that codes through each chapter.
• A sequel repo, Build A Reasoning Model (From Scratch), extends the work with inference-time scaling and RL.

Caveats

• The README is upfront that this is an educational project; the resulting model is small and not competitive with production LLMs.
• It is tightly coupled to the Manning book, so the repository reads like a course syllabus rather than a standalone framework.

Verdict Grab this if you want to stop treating LLMs as black boxes and actually watch the logits flow. Skip it if you are looking for a drop-in replacement for transformers or a recipe for training a production-grade model at scale.