Pruning LLMs by letting activations vote on which weights matter

What it does

Wanda is a one-shot pruning technique for large language models. Instead of stripping weights based only on how small they are, it scores each weight by multiplying its magnitude by the norm of the corresponding input activation, then prunes on a per-output basis. The repository provides PyTorch code to apply this to LLaMA, LLaMA-2, and OPT models, supporting both unstructured and structured N:M sparsity patterns like 2:4 and 4:8.

The interesting bit

The core insight is that a weight’s importance depends on how much activation actually flows through it. A large weight on a dead channel is less valuable than a medium weight carrying heavy traffic. The README’s LLaMA-2 results show this simple product metric often edges out magnitude pruning and sometimes matches SparseGPT on perplexity, particularly at 50% unstructured sparsity.

Key highlights

• One-shot pruning with no retraining required for the base method.
• Supports both unstructured and structured N:M sparsity (2:4 and 4:8).
• Evaluated on LLaMA, LLaMA-2, and OPT; includes zero-shot evaluation support.
• Provides optional ablation code for OBS weight updates and LoRA fine-tuning scripts.
• Built on top of the SparseGPT codebase.

Caveats

• Zero-shot evaluation requires manually downloading and installing a custom lm_eval package from a Google Drive link pinned to a specific commit, which is awkward.
• Details on pruning image classifiers are buried in a subdirectory with no explanation in the main README.

Verdict

A solid stop for researchers benchmarking LLM sparsity, especially on LLaMA-family models. Less useful if you need a polished, general-purpose compression library with a stable evaluation API.