An RNN that claims to match transformers, now in chatbot form

What it does

ChatRWKV is the reference chat interface for RWKV, a recurrent neural network language model that claims to match transformers on quality and scaling while using less VRAM. The repo provides demo scripts, a v2 chat engine with streaming and model-splitting strategies, and reference implementations—including a full model in 150 lines of Python—for running inference and multi-turn conversation.

The interesting bit

The project treats the RNN state as a first-class citizen, which is both the source of its efficiency and a foot-gun: the README repeatedly warns that you must track the text corresponding to each state manually or you will introduce subtle bugs. It also squeezes large models into modest hardware; the authors note that 3 GB of VRAM is enough to run a 14B model in INT8 mode.

Key highlights

• 100% RNN architecture that claims parity with transformers on quality and scaling
• v2 inference supports “stream” and “split” strategies plus INT8, aiming for 3 GB VRAM usage on 14B models
• Includes minimal educational implementations: RWKV in 150 lines and v5 in 250 lines
• Active ecosystem with community inference engines (CPU, Vulkan, CUDA), fine-tuning tools, and mobile apps
• Latest RWKV-7 preview models available with arXiv preprint

Caveats

• State management is manual and fragile; the README warns against calling forward() directly and insists you track text-to-state mappings yourself to avoid bugs
• Chat formatting is rigid and version-dependent (v4-raven expects Bob/Alice, v4/v5/v6-world expects User/Assistant) with strict newline rules
• Getting CUDA kernel acceleration requires compiling a custom kernel; the repo nudges you toward community projects for the fastest inference paths

Verdict

Worth a look if you are curious about non-transformer LLM architectures or need to run large models on constrained VRAM. Skip it if you want a polished, turn-key chat UI—this is a reference implementation that expects you to manage state and formatting yourself.