← all repositories

rasbt/LLM-workshop-2024

An educational workshop teaching how to implement and pretrain large language models from the ground up in PyTorch.

1.1k stars Jupyter Notebook LearningLanguage ModelsML Frameworks
LLM-workshop-2024
Velocity · 7d
+1.5
★ / day
Trend
steady
star history

This repository provides a structured coding workshop for understanding how LLMs work under the hood. Participants code a small GPT-like model from scratch, including the data pipeline, core transformer architecture components, and pretraining loop. The workshop also covers loading pretrained weights and finetuning LLMs using the LitGPT library. The material is based on the book Build a Large Language Model From Scratch.

heatdrop uses Google Analytics to see which pages get read — nothing else. Your call. How we handle data.