Drop-in gateway that compresses coding-agent prompts with a 4B model

What it does

Paritok is a transparent proxy that sits between coding agents — Claude Code, Cursor, Codex, OpenHands — and their upstream LLM. On every request it rewrites the prompt to strip redundant tool schemas, compress file reads and tool outputs down to roughly a quarter of their original size using a dedicated 4B model, and summarize stale history. Because the compression is non-destructive, the agent can call read_original to pull back exact bytes on demand.

The interesting bit

The biggest single-turn win is not the neural compression but an embedding-based semantic filter that drops irrelevant tool schemas from roughly 29K to 8K tokens while keeping the tools[] block byte-stable for prompt caching. The 4B model, distilled on 45K real trajectories atop Qwen3-4B, handles the content layer — protecting identifiers, paths, and error strings while dropping noise. The project claims savings climb from ~25% on turn one to past 60% across long sessions and past 85% in saturated deployments, with the compounding effect coming from compressed file reads staying compressed in history turn after turn.

Key highlights

• Three independent levers: an embedding-based tool-schema filter (biggest per-turn cut), 4B semantic content compression down to ~26% of original size, and history summarization.
• Tool-schema filter runs a local CPU embedding model (BAAI/bge-small-en-v1.5) and keeps the block cache-stable so it never invalidates the LLM’s KV cache.
• The 4B compression model is trained on 45K teacher-distilled coding-agent trajectories and is fully open-source under Apache 2.0.
• Non-destructive design: compressed segments are tagged [REF:id] and the exact original is recoverable instantly via read_original without extra round-trips.
• Drop-in deployment: agents point at Paritok via BASE_URL; no agent code changes required.

Caveats

• The public benchmark is a single read-only “find the bug” task; write-heavy or multi-file refactoring workloads aren’t shown.
• Most of the turn-one saving comes from the embedding-based tool filter, not the 4B model — the neural compression only compounds meaningfully after several turns.

Verdict

Worth a look if you run Claude Code, Cursor, or Codex daily and pay per input token — especially if your sessions run long enough for history to bloat. If your agent already exposes only a handful of tools or you bill by output, the savings will be thinner.