Steer LLM Generation with Sandboxed Wasm Controllers

What it does

AICI is an inference-stack layer that lets you write Controllers—lightweight Wasm programs—to intercept and shape LLM output token-by-token. Instead of relying solely on prompt engineering, you send code (Python, JavaScript, Rust, etc.) that executes on the CPU while the GPU generates text, enforcing formats, stopping conditions, or multi-step logic mid-generation. It abstracts the underlying inference engine so the same controller can run across llama.cpp, HuggingFace Transformers, and rLLM.

The interesting bit

The architecture treats the LLM less like a chat endpoint and more like a coprocessor you orchestrate: the controller can fix tokens, label positions, branch into parallel generations, and maintain state across the request. Because controllers are sandboxed Wasm modules compiled to native code, they run safely without filesystem or network access, making the approach fast enough for tight inference loops and safe enough for eventual multi-tenant cloud deployments.

Key highlights

• Controllers compile to Wasm and run on the CPU in parallel with GPU inference, adding minimal overhead.
• Supports multiple languages: write controllers in Rust, C, C++, or send Python/JavaScript scripts with each request.
• Integrates with llama.cpp, HuggingFace Transformers, and rLLM; vLLM support is noted as in the works.
• Designed to host higher-level control libraries like Guidance and LMQL underneath for portability.
• Sandboxed execution means controllers cannot access the filesystem, network, or other host resources.

Caveats

• The project is explicitly labeled a prototype from Microsoft Research.
• Native Windows support is missing; WSL2 or devcontainer is required.
• The README notes that if you only need constrained decoding, the actively maintained LLGuidance library is the recommended specialization instead.

Verdict

Worth a look if you are building inference infrastructure or experimenting with structured generation and want programmable control inside the decoding loop. Skip it if you just need a production-ready constrained-decoding library today.