Reverse-engineering Claude Mythos as a trainable looped transformer

What it does

OpenMythos implements a Recurrent-Depth Transformer—also called a Looped Transformer—in PyTorch. It splits the model into a Prelude, a recurrent middle block that can be looped up to max_loop_iters times per forward pass, and a Coda. The attention layer is swappable between Grouped Query Attention and Multi-Latent Attention, and the feed-forward network uses a sparse Mixture-of-Experts with routed and shared experts. Pre-configured model sizes range from 1B to 1T parameters, and a training script for the 3B variant on FineWeb-Edu is included.

The interesting bit

The project treats the looped transformer as a discrete dynamical system and constrains the learned injection matrix A so its spectral radius stays below 1 by construction, aiming to prevent the residual explosions that typically destabilize recurrent training. The theory is that looping the same weights silently in latent space—without emitting intermediate tokens—acts as an implicit chain-of-thought, allowing depth extrapolation and systematic generalization to emerge from a single forward pass.

Key highlights

• Switchable attention: GQAttention with optional Flash Attention 2, or MLAttention with compressed KV caching
• Sparse MoE feed-forward with configurable routed and shared experts
• Stability mechanism borrowed from the Parcae architecture to keep the recurrent state bounded
• Configurable loop count at inference time, trading compute for effective depth without adding parameters
• Training script supports PyTorch DDP and mixed-precision training on hardware from older GPUs up to H100s

Caveats

• This is explicitly a speculative, unaffiliated reconstruction based on public research and guesswork, not an official or verified implementation of any Anthropic system
• The README truncates mid-sentence while describing the stability fix, leaving the full technical justification incomplete
• Claims about emergent reasoning behaviors (systematic generalization, depth extrapolation) are theoretical and drawn from cited literature rather than demonstrated results in the repository itself

Verdict

Worth a look if you are researching looped transformers, test-time compute scaling, or reverse-engineering rumored architectures. Skip it if you need a production-ready language model with proven training results.