This repository contains the official code for the ReLoRA paper, which proposes training high-rank networks through low-rank update matrices. It implements a PEFT (Parameter Efficient Fine-Tuning) approach specifically designed for LLaMA-style transformer models. The method resets low-rank components periodically and scales learning rates to enable full-rank training through efficient parameter updates, with support for distributed training across multiple GPUs.