Swap till you drop: JittorLLMs runs 7B models on 2GB RAM

What it does

JittorLLMs is a large language model inference library built atop the Chinese Jittor deep-learning framework. It packages several popular open-weight models—including ChatGLM, LLaMA/LLaMA2, ChatRWKV, PanGu, and Atom7B—and exposes them through a CLI, a browser-based Gradio UI, and a lightweight HTTP API. The project is aimed squarely at lowering hardware barriers, claiming to cut deployment costs by roughly 80% compared with comparable frameworks.

The interesting bit

Its core gimmick is dynamic tensor swapping: the runtime automatically shuffles data between VRAM, system RAM, and disk during inference, which the authors claim is the first implementation for dynamic computation graphs that requires no code changes. They also provide a PyTorch-compatible shim called JTorch, meant to let existing Hugging Face or Megatron scripts run on Jittor’s runtime unmodified while benefiting from the framework’s memory optimizations.

Key highlights

• Claims to run inference with only 2GB of RAM and no GPU, though you still need about 40GB of free disk space for weights and swap files.
• Supports a mix of Chinese-centric models (ChatGLM, PanGu, Atom7B) and English-centric models (LLaMA, LLaMA2, ChatRWKV).
• Zero-copy model loading is claimed to reduce load-time overhead by 40% versus PyTorch, while meta-operator auto-compilation is said to improve compute performance by over 20%.
• Ships with CLI, Gradio web, and HTTP API front ends out of the box.
• Runs on Windows, macOS, and Linux.

Caveats

• MOSS is listed among supported models, but the README immediately redirects users to the official MOSS repository and keeps MOSS in a TODO list, suggesting integration is still pending.
• The advertised “2GB RAM, no GPU” figure is the absolute floor; the same page recommends 32GB of RAM and 16GB of VRAM for comfortable operation.
• Performance claims—40% faster loading and 20% better compute than “similar frameworks”—are stated without disclosed methodology or reproducible benchmarks.

Verdict

Give it a spin if you are hardware-constrained, curious about non-PyTorch Chinese frameworks, or need to squeeze LLMs onto edge devices where memory is tighter than patience. If you already have a high-end GPU and a stable PyTorch pipeline, this is likely more experimental than essential.