LLM output so constrained it skips its own forward passes

What it does Guidance lets you script large language models with ordinary Python idioms—with blocks, string concatenation, and function composition—while enforcing hard constraints on what the model can emit. You can lock generation to a regex, a context-free grammar, a JSON schema, or a multiple-choice list, and interleave that generation with standard control flow like loops and conditionals. It wraps multiple backends (Transformers, llama.cpp, OpenAI, and others) behind an immutable model object that you manipulate like a string builder.

The interesting bit The library treats the LLM as a token-level engine rather than a chat endpoint, so when a grammar makes the next few tokens obvious—say, the closing h1> after an opening tag—Guidance “fast-forwards” them without burning a GPU forward pass. That constraint-driven optimization is the quiet payoff behind the Pythonic syntax.

Key highlights

• Constrain generation with regex, select() lists, or full context-free grammars composed via the @guidance decorator.
• Generate valid JSON by passing a Pydantic model directly to the gen_json helper.
• Debug grammars offline using the Mock model and local match() validation—no API keys required.
• Immutable model objects mean you branch conversations by simple assignment instead of mutating state.
• Ships with a Jupyter widget that visualizes token-by-token generation in real time.

Caveats

• Not every backend supports the full constraint stack; the README warns that CFG features depend on the LLM backend having “full support for Guidance.”
• The learning curve sits somewhere between prompt engineering and writing a parser; you need to think in grammars, not just strings.

Verdict Reach for Guidance if you are building agents, data extractors, or structured-output pipelines where token-level control matters more than chatty flexibility. If you just need a quick Q&A wrapper around an API, it is probably overkill.