MuLabPKU/PiSSA
PiSSA is a parameter-efficient fine-tuning method for large language models that optimizes principal singular values and vectors for faster convergence and better performance than LoRA.
Not currently ranked — collecting fresh signals.
star history
PiSSA (Principal Singular values and Singular vectors Adaptation) is a PEFT method for LLMs that optimizes the essential singular values and vectors while freezing the noisy parts, in contrast to LoRA which freezes the original matrix. The technique achieves faster convergence and better performance across benchmarks like GSM8K, surpassing LoRA by up to 5.16% on Mistral-7B. It also integrates with quantization (QLoRA) to reduce quantization error by 18.97% in LLaMA 2-7B and has been merged into the HuggingFace peft library.
Frequently asked
- What is MuLabPKU/PiSSA?
- PiSSA is a parameter-efficient fine-tuning method for large language models that optimizes principal singular values and vectors for faster convergence and better performance than LoRA.
- Is PiSSA open source?
- Yes — MuLabPKU/PiSSA is an open-source project tracked on heatdrop.
- What language is PiSSA written in?
- MuLabPKU/PiSSA is primarily written in Jupyter Notebook.
- How popular is PiSSA?
- MuLabPKU/PiSSA has 428 stars on GitHub.
- Where can I find PiSSA?
- MuLabPKU/PiSSA is on GitHub at https://github.com/MuLabPKU/PiSSA.