A local inference server that actually understands agents

What it does Locals is a local model server for macOS on Apple silicon, built specifically for agent workloads rather than chat. It speaks the OpenAI, Anthropic and Ollama APIs, so OpenCode, Cline, Aider, OpenHands and Claude Code can point at it without changes. Everything runs on your machine — nothing leaves it.

The interesting bit It’s grounded in a paper (“Breaking the Space Barrier…”) that treats JSON formats and tool calls as graphs: text the format already determines gets written without running the model at all, and schema checks run on the GPU in roughly 3 MB instead of up to 1.5 GB. Tool calls always parse, arguments match their schema types, and a reply cut off mid-call still closes it — the classic small-model failure mode, engineered away.

Key highlights

• Concurrent requests share GPU kernels: 8 agents at once hit 271 tokens/s vs mlx-lm’s 87 (3.1x, M2 Pro, Qwen3.5-4B)
• Conversations persist across turns and restarts; a follow-up on a 48,000-token document answers in 0.6 s where mlx-lm ran out of memory
• Guards against repetitive loops, endless tool calls, and out-of-memory errors when many agents run together
• locals run splits a task across multiple agents, each in its own project copy, with your tests as the referee
• Ships kernels for newer models (Liquid’s LFM3, Recursive’s Flint) that mlx-lm can’t load yet

Caveats

• Apple silicon only; M1, M3 and M4 chips are unmeasured, and gains shrink on the M5 Pro where mlx-lm is already fast
• Flint needs 24 GB+ of RAM and isn’t on PyPI yet at time of writing
• Benchmarks are mostly self-reported on the authors’ own hardware

Verdict If you run coding agents locally on a Mac and have hit mlx-lm’s concurrency ceiling, this is worth a serious look. If you’re on Linux, Windows, or just chatting with one model, it’s not for you.