AGI-Arena/MARS
MARS is an optimizer framework that combines variance reduction with preconditioned updates to accelerate training of large language models.
Not currently ranked — collecting fresh signals.
star history
MARS (Make Variance Reduction Shine) provides a unified optimization framework addressing gradient variance challenges in large model training. It implements a scaled stochastic recursive momentum for variance-reduced gradient estimation combined with a preconditioned update approximating second-order Newton methods. The implementation includes CUDA kernels and supports both pretraining (GPT-2 XL, FineWeb-Edu) and fine-tuning workflows for large language models.
Frequently asked
- What is AGI-Arena/MARS?
- MARS is an optimizer framework that combines variance reduction with preconditioned updates to accelerate training of large language models.
- Is MARS open source?
- Yes — AGI-Arena/MARS is open source, released under the Apache-2.0 license.
- What language is MARS written in?
- AGI-Arena/MARS is primarily written in Python.
- How popular is MARS?
- AGI-Arena/MARS has 723 stars on GitHub.
- Where can I find MARS?
- AGI-Arena/MARS is on GitHub at https://github.com/AGI-Arena/MARS.