Guitaricet/relora
Official implementation of ReLoRA, a parameter-efficient fine-tuning method for training LLaMA models with low-rank updates.
Not currently ranked — collecting fresh signals.
star history
This repository contains the official code for the ReLoRA paper, which proposes training high-rank networks through low-rank update matrices. It implements a PEFT (Parameter Efficient Fine-Tuning) approach specifically designed for LLaMA-style transformer models. The method resets low-rank components periodically and scales learning rates to enable full-rank training through efficient parameter updates, with support for distributed training across multiple GPUs.
Frequently asked
- What is Guitaricet/relora?
- Official implementation of ReLoRA, a parameter-efficient fine-tuning method for training LLaMA models with low-rank updates.
- Is relora open source?
- Yes — Guitaricet/relora is open source, released under the Apache-2.0 license.
- What language is relora written in?
- Guitaricet/relora is primarily written in Jupyter Notebook.
- How popular is relora?
- Guitaricet/relora has 474 stars on GitHub.
- Where can I find relora?
- Guitaricet/relora is on GitHub at https://github.com/Guitaricet/relora.