Coding agent built for models that fit on your GPU

What it does

SmallCode is a terminal-native AI coding agent built for local LLMs in the 8B–35B parameter range. It assumes your model has limited context, unreliable tool calling, and a tendency to loop or forget tasks, then compensates with strict context budgets, semantic history compression, and early-stop detection for patch spirals and repetition. The agent runs fully offline against local endpoints, with an optional opt-in escalation to cloud APIs when the local model hard-fails after retries.

The interesting bit

The project treats small-model fragility as a systems engineering problem. Its MarrowScript layer lets you declare a 50-line prompt specification that compiles to 1,400+ lines of TypeScript complete with caching, retry logic, structured traces, and token-budget enforcement—essentially generating reliability infrastructure from a DSL. For Node.js backends, BoneScript collapses entire project scaffolding into a single compiled file, shortening the long tool-call chains that typically break small models.

Key highlights

• 2-stage tool routing halves schema overhead by selecting a category first, then exposing only relevant tool definitions
• Hybrid code search mixes BM25 exact matching with hashed-vector semantic ranking, running entirely offline on CPU
• Forgiving tool-call parser accepts JSON, YAML, XML, Hermes, Liquid AI, or plain text, auto-repairing parameter mistakes
• Context budget engine caps tool results at 4k characters and evicts or compresses history mid-turn to prevent window overflow
• Optional tiered escalation to Claude, OpenAI, or DeepSeek endpoints, session-limited to prevent runaway cloud costs

Caveats

• The repository description claims an “87% benchmark with 4B-active model,” but the README explicitly warns that models ≤4B struggle with multi-step tool use and recommends 8B–35B instead
• better-sqlite3 requires either prebuilt binaries or a C++ toolchain; if compilation fails, the agent falls back to JSON-based memory automatically
• A transitive dependency emits an unmaintained-package warning that the authors note cannot currently be silenced

Verdict

Developers running local 8B–35B models who are tired of watching frontier-designed agents choke on consumer hardware should try this; if you already have reliable API access to Claude or GPT-5, SmallCode’s compensations are likely unnecessary.