Constrained deep learning without the differential-geometry PhD

What it does

GeoTorch is a PyTorch library that pins neural-network weights to geometric constraints—orthogonal columns, fixed or bounded rank, positive definiteness, and more—by wrapping existing nn.Module parameters. You declare the constraint when building the model, and the library handles the projection during forward passes while standard optimizers update the underlying free parameters behind the scenes.

The interesting bit

Rather than forcing you to write custom Riemannian optimizers, GeoTorch treats each constraint as a differentiable map from a flat parameter space onto a manifold, so Adam, SGD, or any PyTorch optimizer can still drive training. If you do want true Riemannian gradient descent, the raw manifolds are exposed for that too.

Key highlights

• One-line decorators such as geotorch.orthogonal(layer, "weight") or geotorch.low_rank(layer, "weight", rank=1) retrofit constraints onto existing layers.
• Covers a wide catalog: orthogonal, low-rank, fixed-rank, symmetric, skew-symmetric, positive (semi)definite, invertible, unit-determinant, sphere, Grassmannian, and product manifolds.
• Supports batched tensor shapes (*, n, k) for efficient parallel product-space operations.
• Factorized spaces expose singular values, making determinant computations straightforward for normalizing-flow building blocks.
• Compatible with PyTorch >= 2.6 and Python >= 3.10; tested on Linux, macOS, and Windows.

Caveats

• Repeated use of a parametrized layer (for example inside an RNN) requires wrapping the forward pass with parametrize.cached() to avoid recomputing the constraint each call—an easy detail to miss.
• It is fundamentally a parametrization layer atop PyTorch’s nn.utils.parametrize; the heavy lifting is the manifold math, not a new training framework.
• Requires PyTorch >= 2.6, so legacy codebases may need to upgrade first.

Verdict

Researchers and practitioners who need structured layers—orthogonal recurrent weights, low-rank convolutions, or PSD covariance heads—without rewriting their training loop should take a look. If your model is already unconstrained and training fine, this is probably overkill.