Om-Alve/smolGPT
A minimal PyTorch implementation for training a small GPT language model from scratch with modern architecture features.
Not currently ranked — collecting fresh signals.
star history
This repository provides a from-scratch implementation of a GPT model in pure PyTorch with no abstraction overhead. It includes modern architectural components such as flash attention, RMSNorm, SwiGLU activation, and optional rotary embeddings (RoPE). The project supports the full training pipeline including mixed precision training, gradient accumulation, warmup scheduling, and a built-in TinyStories dataset processor with SentencePiece tokenizer integration.
Frequently asked
- What is Om-Alve/smolGPT?
- A minimal PyTorch implementation for training a small GPT language model from scratch with modern architecture features.
- Is smolGPT open source?
- Yes — Om-Alve/smolGPT is open source, released under the MIT license.
- What language is smolGPT written in?
- Om-Alve/smolGPT is primarily written in Python.
- How popular is smolGPT?
- Om-Alve/smolGPT has 1.5k stars on GitHub.
- Where can I find smolGPT?
- Om-Alve/smolGPT is on GitHub at https://github.com/Om-Alve/smolGPT.