← all repositories
hyperionaidev/smolmaxxing

An 80B model on a phone, a 16B on a TV box

smolmaxxing patches llama.cpp until forgotten hardware — TV boxes, a 2013 iPad, old phones — runs models far bigger than its RAM should allow.

★1.1k stars C Inference · Serving
smolmaxxing
Too new to rank. Velocity needs a few days of history.
not enough history yet
star history

What it does

It’s a set of llama.cpp patches plus a pile of measured results, all aimed at one question: what’s the biggest model a given piece of junk hardware can actually run? So far three devices: a Xiaomi TV Box S running 16B MoE models at 5.1 tok/s on ~1 GB of RAM, a jailbroken 2013 iPad Air hitting 24–26 tok/s with a 1-bit 1.7B model, and a OnePlus phone running Qwen3-Next-80B-A3B at 2.6–3.1 tok/s with about 3 GB free. Every number was measured on the device itself, and the repo includes a lab notebook of 165 experiments, failures included.

The interesting bit

The core trick is treating flash as an extension of RAM. For mixture-of-experts models, only a few experts fire per token, so the popular ones stay in RAM and the rest get paged from flash on demand, with the next layer prefetched. That’s how 16B fits in 1 GB and 80B fits on a phone. There’s also vocabulary pruning — most of a model’s output rows are rarely used, so they’re simply dropped.

Key highlights

  • Custom kernels per chip: 1-bit lookup tables, repacked 2-bit and ternary formats for Cortex-A55, and variants for the Apple A7, which lacks dot-product instructions
  • Dense models bigger than RAM stream from flash at the flash’s full speed
  • A “headless” mode on the TV box stops the launcher and Google services while a model runs
  • Ready-made model files on Hugging Face, plus tools to rebuild them
  • The techniques aren’t device-specific — the 1-bit kernel needs only basic NEON, and the pager works on any Linux/Android device

Caveats

  • The TV box and iPad model files need the patched llama.cpp; stock llama.cpp can’t load their pruned heads
  • The iPad path requires a jailbroken device on iOS 12
  • Three devices so far — single-board computers and routers are listed as “coming”

Verdict

Anyone who owns idle hardware with a CPU and some flash should browse the results docs for ideas alone. If you only ever run models on a GPU box with plenty of VRAM, it’s a fascinating read but not a tool you need.

Frequently asked

What is hyperionaidev/smolmaxxing?
smolmaxxing patches llama.cpp until forgotten hardware — TV boxes, a 2013 iPad, old phones — runs models far bigger than its RAM should allow.
Is smolmaxxing open source?
Yes — hyperionaidev/smolmaxxing is open source, released under the MIT license.
What language is smolmaxxing written in?
hyperionaidev/smolmaxxing is primarily written in C.
How popular is smolmaxxing?
hyperionaidev/smolmaxxing has 1.1k stars on GitHub.
Where can I find smolmaxxing?
hyperionaidev/smolmaxxing is on GitHub at https://github.com/hyperionaidev/smolmaxxing.

heatdrop uses Google Analytics to see which pages get read — nothing else. Your call. How we handle data.