The Grammar Engine Inside vLLM, SGLang, and TensorRT-LLM

What it does XGrammar is a C++ library that clamps down on LLM token decoding to enforce structural rules. It uses constrained decoding to guarantee that outputs stick to a given grammar—JSON, regex, or a fully custom context-free grammar—so the model cannot hallucinate a missing bracket or invalid schema field. The project exposes Python, C++, and JavaScript APIs and supports hardware from CPU to TPU.

The interesting bit The speed claim is “near-zero overhead” for JSON, but the real curiosity is how thoroughly XGrammar has infiltrated the inference stack. It is already the default structured generation backend for vLLM, SGLang, TensorRT-LLM, and MLC-LLM, among others, turning one open-source C++ core into shared infrastructure across competing engines. That level of adoption usually requires years of committee meetings; here it appears to have spread by simply being useful.

Key highlights

• Enforces 100% structural correctness via constrained decoding against context-free grammars
• Claims near-zero overhead during JSON generation
• Runs on Linux, macOS, and Windows across CPU, NVIDIA GPU, AMD GPU, Apple Silicon, and TPU
• Provides Python, C++, and JavaScript APIs, plus community Rust bindings
• Default backend in vLLM, SGLang, TensorRT-LLM, MLC-LLM, OpenVINO GenAI, and Modular MAX

Verdict If you self-host LLMs and need machine-readable output that will not break downstream parsers, this is either already in your stack or should be. If you only consume managed APIs, you can safely keep scrolling.