A 28.9-million-parameter LLM that fits on an $8 chip

What it does

This project runs a 28.9 million parameter language model entirely on an ESP32-S3, an ~$8 microcontroller with only 512KB of SRAM. It generates text at about 9.5 tokens per second, displaying each word on a small attached screen, with no network connection required. The model was trained on the TinyStories dataset, so it produces short, coherent children’s-style stories rather than answering questions or writing code.

The interesting bit

The trick is Google’s Per-Layer Embeddings, borrowed from the Gemma models. Instead of loading all 28.9 million parameters into precious SRAM, the 25-million-row embedding table stays in 16MB of flash storage. Each token pulls only about 450 bytes from that table, while the actual “thinking” core remains in fast memory. That makes the model roughly 100× larger than the previous record for a chip like this, without changing the hardware.

Key highlights

• 28.9M parameters total, with 25M living in flash and only ~450B fetched per token
• Runs at ~9.5 tok/s end-to-end on an ESP32-S3 with 512KB SRAM, 8MB PSRAM, and 16MB flash
• Fully offline inference; nothing leaves the device
• 14.9MB model size at 4-bit quantization
• Author preserved the messy commit history, including a corrected bug in early parameter-count claims

Caveats

• The model only knows how to write simple stories from the TinyStories dataset; it cannot answer questions, follow instructions, or write code
• The memory trick expands capacity but does not improve reasoning ability, which is still bound by the small compute core that fits in SRAM

Verdict

Worth a look if you care about squeezing large models into tiny hardware or running inference on devices that lack cloud connectivity. Skip it if you need a general-purpose assistant; this is an architecture demo, not a product.