QwenLM/ParScale
ParScale is a novel LLM scaling paradigm that applies P parallel learnable transformations to inputs, executes model forward passes in parallel, and aggregates outputs to achieve logarithmic scaling comparable to O(log P) parameter growth.
This repository presents a theoretical and empirical framework for scaling language models beyond traditional parameter and inference-time scaling approaches. The method applies P diverse and learnable transformations to the input, executes forward passes in parallel, and dynamically aggregates the P outputs during both training and inference. The work establishes a logarithmic scaling law demonstrating that parallel computation can serve as an efficient substitute for parameter growth in larger models. Pre-trained models and code are provided via Hugging Face.
Frequently asked
- What is QwenLM/ParScale?
- ParScale is a novel LLM scaling paradigm that applies P parallel learnable transformations to inputs, executes model forward passes in parallel, and aggregates outputs to achieve logarithmic scaling comparable to O(log P) parameter growth.
- Is ParScale open source?
- Yes — QwenLM/ParScale is an open-source project tracked on heatdrop.
- What language is ParScale written in?
- QwenLM/ParScale is primarily written in Python.
- How popular is ParScale?
- QwenLM/ParScale has 479 stars on GitHub.
- Where can I find ParScale?
- QwenLM/ParScale is on GitHub at https://github.com/QwenLM/ParScale.