JonasGeiping/cramming
A research framework for training a BERT-style masked language model on a single GPU in one day.
Not currently ranked — collecting fresh signals.
star history
Contains code to replicate research on cramming language model pretraining into limited compute budgets. The framework re-analyzes components of the pretraining pipeline for single-GPU training scenarios and provides a modified pipeline achieving performance close to standard BERT. Supports PyTorch 2.0+ with improved checkpoints available on Hugging Face.
Frequently asked
- What is JonasGeiping/cramming?
- A research framework for training a BERT-style masked language model on a single GPU in one day.
- Is cramming open source?
- Yes — JonasGeiping/cramming is open source, released under the MIT license.
- What language is cramming written in?
- JonasGeiping/cramming is primarily written in Python.
- How popular is cramming?
- JonasGeiping/cramming has 1.4k stars on GitHub.
- Where can I find cramming?
- JonasGeiping/cramming is on GitHub at https://github.com/JonasGeiping/cramming.