PKU-Alignment/safe-rlhf
An RLHF framework for training value-aligned LLMs with safety constraints, developed by Peking University's alignment team.
Not currently ranked — collecting fresh signals.
star history
Beaver is a modular RLHF framework that supports supervised fine-tuning (SFT), RLHF, and Safe RLHF training for popular pre-trained models including LLaMA, OPT, and Baichuan. It provides human-labeled preference datasets (up to 1M pairs) combining helpful and harmless labels, pre-trained reward and cost model checkpoints, and multi-scale safety evaluation metrics. The project implements the Safe RLHF method for constrained value alignment, accepted at ICLR 2024.
Frequently asked
- What is PKU-Alignment/safe-rlhf?
- An RLHF framework for training value-aligned LLMs with safety constraints, developed by Peking University's alignment team.
- Is safe-rlhf open source?
- Yes — PKU-Alignment/safe-rlhf is open source, released under the Apache-2.0 license.
- What language is safe-rlhf written in?
- PKU-Alignment/safe-rlhf is primarily written in Python.
- How popular is safe-rlhf?
- PKU-Alignment/safe-rlhf has 1.6k stars on GitHub.
- Where can I find safe-rlhf?
- PKU-Alignment/safe-rlhf is on GitHub at https://github.com/PKU-Alignment/safe-rlhf.