Server-class models, laptop-sized GPU: PolyStrata's local inference bet

What it does

PolyStrata runs large AI models locally on ordinary hardware and exposes them as an OpenAI/Anthropic-compatible API on localhost, so chat apps and coding agents like Claude Code can talk to them directly. It ships ten models across two engines: Qwen3.8-Flash-Next on the borrowed Strata engine, and nine more (GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Flint, Qwen3.6, Ornith 1.5, Gemma 4, Tiel Coder, MiniCPM-V) on an engine written for this project. Nothing leaves your PC.

The interesting bit

The trick is aggressive memory tiering: the big models (79–114 GB files) mostly live in system RAM with only part locked for the GPU, which is why a 35B model can write at 94–100 tokens per second on a laptop with an 8 GB card. The README also claims prompt reading 2.7–3.5× faster than llama.cpp on that laptop — and publishes a benchmark you can rerun yourself, which is more honesty than most local-LLM projects offer.

Key highlights

• Ten models, several vision-capable, swappable while the server runs
• Serves OpenAI and Anthropic API formats on localhost, so existing agents just work
• Multi-GPU support (two or three cards can share a model)
• One-click install on Windows and Linux; Apple Silicon supported for Qwen3.8-Flash-Next only
• Includes an MCP server so AI assistants can install, start and stop it themselves

Caveats

• The nine models on PolyStrata’s own engine are explicitly experimental — validated on essentially one workstation and one laptop
• GLM 5.3 Flash and DeepSeek V4 Flash need a 32 GB card plus 72–96 GB of RAM, and Linux only
• Model startup can freeze your PC for 1–3 minutes while 35–55 GB loads into RAM; the README warns you not to panic
• LFM2.5’s license restricts commercial use to companies under $10M yearly revenue

Verdict

Worth a look if you want coding agents and chat running on your own hardware with a familiar API, and you have the RAM to feed it. If you’re on an older card, Intel Arc, or hoping for a battle-tested engine, the FAQ’s caveats apply — this is promising, not proven.