An RNN that trains like a transformer and learns at test time

What it does

RWKV is a language-model architecture that claims transformer-level performance while remaining a 100% RNN: no attention mechanism, no KV-cache, and constant memory usage regardless of sequence length. The repo provides training code, reference implementations, and demo scripts for RWKV-7 (“Goose”), along with earlier versions. It is a Linux Foundation project and, according to the authors, already ships inside Windows and Office.

The interesting bit

RWKV-7 does something odd for an RNN: it performs what the authors call “in-context gradient descent” at every token, effectively test-time-training its hidden state on the fly. That makes it a “meta-in-context learner” rather than a static state machine.

Key highlights

• Linear-time, constant-space inference with no KV-cache; speed and VRAM stay flat as context grows.
• Training parallelizes like a GPT transformer, and the reference implementation fits on a single GPU with 7 GB VRAM for testing.
• Reported RTX 5090 benchmarks for the 7.2B model include 145+ tokens/s single-sequence decode and over 10,000 tokens/s batched decode.
• Training throughput is cited at roughly 260k tokens/s on 4×8 H100s with DeepSpeed.
• Ecosystem includes mobile inference libraries, GGUF weights, Gradio and WebGPU demos, and 400+ community projects.

Caveats

• The third-party FLA implementation is explicitly noted as not yet aligned with the reference, and produces worse performance.
• RWKV-7 is a holistic model, not a reusable layer: initialization, weight decay, and learning rates are tuned per parameter, so dropping a “RWKV-7 block” into another architecture is unsupported.
• The README warns that weight decay must be restricted to large matrix parameters; applying it everywhere degrades performance.

Verdict

Worth exploring if you research efficient sequence models or need an LLM that runs with fixed memory on long contexts. Skip it if you want a plug-and-play PyTorch layer for an existing transformer stack.