princeton-nlp/MeZO
A memory-efficient zeroth-order optimizer that fine-tunes language models using only forward passes, reducing memory usage by up to 12x.
Not currently ranked — collecting fresh signals.
star history
MeZO adapts classical zeroth-order SGD to operate in-place for language model fine-tuning, enabling training of 30B parameter models on a single 80GB GPU (vs 2.7B with Adam). The method achieves comparable performance to backpropagation-based fine-tuning across multiple tasks and supports both full-parameter and parameter-efficient tuning techniques such as LoRA and prefix tuning. It also enables optimization of non-differentiable objectives like accuracy or F1 scores.
Frequently asked
- What is princeton-nlp/MeZO?
- A memory-efficient zeroth-order optimizer that fine-tunes language models using only forward passes, reducing memory usage by up to 12x.
- Is MeZO open source?
- Yes — princeton-nlp/MeZO is open source, released under the MIT license.
- What language is MeZO written in?
- princeton-nlp/MeZO is primarily written in Python.
- How popular is MeZO?
- princeton-nlp/MeZO has 1.2k stars on GitHub.
- Where can I find MeZO?
- princeton-nlp/MeZO is on GitHub at https://github.com/princeton-nlp/MeZO.