PyTorch models can finally explain themselves

What it does

Captum is an interpretability library for PyTorch. It collects general-purpose implementations of attribution methods—Integrated Gradients, saliency maps, SmoothGrad, and others—into one package that plugs into standard PyTorch models and domain-specific libraries like torchvision. It also includes tools for concept-based analysis (TCAV), influence functions (TracIn), and adversarial perturbations for counterfactual explanations.

The interesting bit

Most interpretability libraries are either research prototypes or opaque cloud services. Captum tries to be both a research workbench—researchers can benchmark new algorithms against its built-in baselines—and a production troubleshooting layer, helping engineers explain specific predictions to end users.

Key highlights

• Ships with feature-attribution staples (IntegratedGradients, DeepLift, GradientShap) plus newer methods like TCAV and TracIn influence functions.
• Includes adversarial-attack and minimal-perturbation utilities for counterfactual explanations, going beyond simple post-hoc attribution.
• Integrates with domain-specific PyTorch ecosystems (torchvision, torchtext) without forcing model rewrites.
• Exposes convergence deltas and approximation-error proxies so you can sanity-check whether an attribution is mathematically sound.

Caveats

• The DeepLift implementation currently supports only the Rescale rule and not all non-linear activation types.
• Requires Python >= 3.10 and PyTorch >= 2.3, so older pipelines may need upgrading.

Verdict

Useful if you are building, debugging, or auditing PyTorch models and need to justify predictions to users or stakeholders. Not relevant if you are outside the PyTorch ecosystem or have no use for model explanations.