How structured outputs work: constrained decoding, grammars and JSON Schema

Key points

• A model writes one token at a time: logits, softmax, sample, repeat. One bad token is enough to break a JSON document.
• Constrained decoding masks every token the grammar does not allow at the current position, so invalid output can never be sampled.
• Schemas compile to a finite-state machine when nesting is bounded and to a pushdown automaton when they recurse; engines precompute which tokens are valid in which state to keep each step fast.
• Tokens and grammar symbols do not line up, and that gap is where most of the engineering, and some of the quality loss, lives.
• OpenAI, Anthropic and llama.cpp each support a different subset of JSON Schema. Check the limits before you design the schema.
• Research disagrees on whether constraints hurt answers, so put reasoning before the answer, validate every result and measure on your own data.

Sampling: logits, softmax and the token loop

A language model does not write JSON, or anything else, in one go. It produces one token at a time. A token is a piece of text from a fixed vocabulary: a word, part of a word, a punctuation mark or a run of spaces. GPT-2’s vocabulary, the example used in the Outlines paper, has 50,257 tokens; many newer models use larger ones. Each step of generation runs the same loop:

1. The model reads the prompt and everything generated so far and outputs a vector of logits: one raw score for every token in the vocabulary.
2. Softmax turns the scores into probabilities. Each logit is divided by the temperature, exponentiated and divided by the sum over all tokens. A low temperature sharpens the distribution; near zero, the top token almost always wins.
3. Optional filters such as top-k or top-p (nucleus) sampling drop the unlikely tail and renormalize what is left.
4. One token is sampled and appended to the sequence.
5. The loop repeats until the model emits an end-of-sequence token, hits a stop sequence or reaches the output limit.

Nothing in that loop knows about JSON. The model has learned that { is usually followed by ", so well-formed output is likely, but every token is still a draw from a distribution. The loop is also why long outputs are slow and why serving engines keep a key-value cache of earlier tokens; see prompt caching and the KV cache.

Why asking for JSON is not enough

A clear prompt with an example gets valid JSON most of the time. At volume, "most of the time" becomes a steady trickle of failures, each caused by one unlucky token:

• A preamble such as "Here is the JSON:", or a Markdown code fence around the object.
• A trailing comma, a single-quoted key or an unescaped quote inside a string value.
• A missing required field, an extra field, or the string "2" where the schema wants the integer 2.
• An enum value that is almost right: "High" instead of "high", or a category that is not on the list.
• Output cut off at the token limit, which leaves an object that never closes.

JSON mode, where a provider offers it, fixes the first two problems but not the rest. OpenAI’s docs put it plainly: both JSON mode and Structured Outputs produce valid JSON, but only Structured Outputs makes the output follow your schema. The same docs warn that in JSON mode you still have to tell the model to produce JSON, or it may generate whitespace until it reaches the token limit. Constrained decoding attacks the problem at the source: it takes the bad tokens out of the draw.

Constrained decoding: a mask at every step

The idea fits in one sentence. Between computing the logits and sampling, set the logit of every token that cannot legally come next to minus infinity. After softmax those tokens have probability zero, and the probability mass is spread over the valid tokens in proportion to what the model already preferred.

To know what can legally come next, the decoder keeps a parser state for the output so far. Each step asks the parser which tokens are allowed, masks the rest, samples, and advances the parser with the chosen token. Here is the whole mechanism with a toy vocabulary of eight tokens and a format with exactly two valid outputs:

import math, random

VOCAB = ['{', '}', '"ok"', ':', 'true', 'false', 'yes', '<eos>']
FSM = {  # parser state -> {allowed token: next state}
    0: {'{': 1}, 1: {'"ok"': 2}, 2: {':': 3},
    3: {'true': 4, 'false': 4}, 4: {'}': 5}, 5: {'<eos>': 6},
}

def fake_logits(text):  # stands in for the model's forward pass
    return [random.gauss(0, 2) for _ in VOCAB]

def softmax(xs):
    top = max(xs)
    exps = [math.exp(x - top) for x in xs]  # exp(-inf) == 0.0
    return [e / sum(exps) for e in exps]

def generate():
    text, state = '', 0
    while state != 6:
        allowed = FSM[state]
        logits = fake_logits(text)
        masked = [l if t in allowed else -math.inf for t, l in zip(VOCAB, logits)]
        token = random.choices(VOCAB, weights=softmax(masked))[0]
        text, state = text + token, allowed[token]
    return text.removesuffix('<eos>')  # {"ok":true} or {"ok":false}, never "yes"

The fake model is free to give yes the highest score; it can never be sampled, because no state allows it. A real engine works the same way with a vocabulary of tens of thousands of tokens and a parser generated from your schema. Inference libraries expose the hook directly: in Hugging Face Transformers a logits processor receives the scores for the whole vocabulary at each step and returns modified scores, and the built-in token-suppressing processors set blocked tokens to minus infinity. Because the mask is applied before sampling, it works with greedy decoding, temperature and top-p alike.

The expensive part is computing the mask. The naive approach checks every vocabulary token against the parser at every step, which Willard and Louf describe as a fixed O(N) cost per generated token, where N is the vocabulary size. Most of the research is about making that step cheap.

From JSON Schema to an automaton

A JSON Schema, a regular expression and a grammar are all ways to describe a set of valid strings. Engines compile them into an automaton they can step through one token at a time.

• Finite-state machine (FSM). A regular expression compiles to an FSM: a set of states with transitions on characters. A schema without recursion, where nesting depth is bounded, describes a regular language, so it can be expanded into one large regular expression and handled the same way.
• Pushdown automaton (PDA). Recursive structures, such as a comment whose replies contain comments, or arbitrary JSON, need a stack to match every opening brace with its closing one. That takes a context-free grammar, and a PDA is, roughly, an FSM with a stack.

Willard and Louf’s paper, which the Outlines library implements, reframes generation as transitions between the states of an FSM and precomputes an index from each state to the vocabulary tokens that are valid in it. Building the index walks every token’s full string through the FSM, starting from every state, once, before generation. At generation time the mask is a lookup, which the authors note costs O(1) on average with a hash map, paid for with memory that grows with the number of states. The same paper extends the approach to context-free grammars with LALR(1) parsers and pushdown automata, where the index also has to account for the parser’s stack.

XGrammar (Dong et al.) targets the context-free case. Its authors split the vocabulary into context-independent tokens, whose validity can be decided ahead of time from the position in the grammar, and context-dependent tokens, which have to be checked against the stack at runtime. They transform the grammar to shrink the second group, use a persistent stack to speed up the runtime checks, and overlap grammar work with GPU computation. They report up to a 100x speedup over existing solutions and near-zero overhead for structured generation in end-to-end serving.

llama.cpp shows what a grammar looks like to the user. Its GBNF format defines rules with ::=, alternatives with | and repetition with *, +, ? and counted ranges, and the whole output has to match the root rule. A simplified fragment of JSON:

root   ::= object
value  ::= object | array | string | number | "true" | "false"
object ::= "{" ws ( string ":" ws value ( "," ws string ":" ws value )* )? "}" ws
array  ::= "[" ws ( value ( "," ws value )* )? "]" ws
string ::= "\"" [^"\\]* "\"" ws
number ::= "-"? [0-9]+ ( "." [0-9]+ )? ws
ws     ::= [ \t\n]{0,4}

value refers to object and array, which refer back to value. That recursion is what needs a stack. The fragment leaves out escapes and a few literals, and caps whitespace for a reason covered below. You rarely write one by hand: llama.cpp converts a JSON Schema to GBNF when you pass it with --json on the command line or in the json_schema field of a server request. Its docs note that the schema constrains the output but is not added to the prompt, so describe the structure in the prompt as well, and that unsupported schema features are skipped silently, so inspect the generated grammar.

Where tokens and grammars disagree

Grammars are written over characters; models emit tokens. The two rarely line up, and much of the hard engineering lives in that gap.

• One token can span several grammar symbols. A tokenizer may have a single token for ": or "},, which ends a string and moves on to the next part of the object in one step. The engine may accept that token only if every one of its characters is valid in sequence, which is why the Outlines index walks each token’s whole string through the FSM, starting from every state.
• A valid string can force an unusual split. When the constraint leaves only one possible continuation, such as a fixed field name, the engine may emit it with a tokenization the model rarely saw in training, and the next token is predicted from an odd context.
• The prompt boundary has the same problem. The Guidance docs show a prompt ending in http: where the model continued with a space and an invalid URL, because the colon on its own is a rare token and :// is normally a single one. Their fix, token healing, backs up one token and constrains the first generated token to start with the text that was removed.
• Whitespace is a trap. JSON allows any amount of it between values, so a grammar that permits unlimited whitespace lets a confused model emit newlines until the token limit. Capping it in the grammar, as the fragment above does, or fixing it to a single space closes that door.

These effects are one reason constrained generation can change answers, not only their format. The authors of JSONSchemaBench suggest that the accuracy gains they measured for the Guidance engine may partly come from its token healing.

What provider strict modes support

Hosted APIs run constrained decoding for you when you set strict: true on a tool or ask for a JSON Schema response format. Each supports a subset of JSON Schema, and the subsets differ. As documented in November 2026:

Three behaviors matter more than the table:

• Refusals and truncation escape the schema. OpenAI returns a refusal as its own content item instead of schema-shaped output. Anthropic returns stop_reason: "refusal" and says the output might not match the schema, and with stop_reason: "max_tokens" the output may be incomplete. Check for these before parsing.
• Order is part of the format. OpenAI generates keys in the order of the schema. Anthropic puts required properties first and optional ones after, so mark everything required if the order matters. It matters for quality, as the next section shows.
• Schemas are compiled and cached. OpenAI warns that the first request with a new schema is slower. Anthropic compiles a grammar on first use, caches it for 24 hours from last use and recompiles when the schema structure changes; changing only names or descriptions does not invalidate it.

Anthropic also documents a sharp edge: the model may change the capitalization of enum and const values with no error, so compare them case-insensitively. Strict mode for tool arguments is covered in how LLM tool calling works, and the product-level view is in LLM structured outputs.

Does constraining the output hurt quality?

Masking changes the distribution the model samples from, so it can change the answers. The published evidence points both ways.

• Tam et al., "Let Me Speak Freely?" (2024), compared constrained JSON mode, format instructions in the prompt, and a two-step method that answers in natural language first and converts the answer to the format afterwards. On reasoning tasks such as GSM8K and Last Letter Concatenation, the authors report that stricter format constraints generally caused larger drops, with JSON mode usually worst. On classification tasks JSON mode was competitive and sometimes best, which they attribute to a narrower answer space. In one task the constrained output put the "answer" key before the "reason" key, so the model answered before it reasoned.
• Geng et al., "JSONSchemaBench" (2025), tested six engines, among them Guidance, Outlines, llama.cpp, XGrammar, OpenAI and Gemini, on 10,000 real-world schemas. The authors report that constrained decoding improved accuracy over unconstrained generation by up to 4% on the reasoning tasks they tried, and that no engine covered all of JSON Schema: the best supported about twice as many of their schemas as the worst.

The practical reading: constraints are cheap insurance for extraction and classification, and the risk sits in tasks where the model has to think inside the output. Three habits help:

• Put reasoning before the answer. A short reasoning field ahead of the result avoids the ordering Tam et al. found harmful. Anthropic’s docs warn that a field asking for the model’s full reasoning can trigger a refusal and suggest asking for a short explanation instead.
• Separate thinking from formatting. Let the model reason in free text, or in a provider’s thinking mode, and produce the structured result last. Anthropic notes that its grammar applies to the final output, not to thinking.
• Measure on your own data. Run the same eval set with and without constraints before you decide; how to evaluate RAG covers building one.

Validate anyway: retries and fallbacks

Constrained decoding guarantees shape, within the supported subset. It does not guarantee that the content is right, and some outputs, such as refusals and truncated replies, escape it entirely. Keep a validator in the path. A schema for triaging support tickets:

{
  "type": "object",
  "properties": {
    "reasoning": { "type": "string", "description": "One or two sentences" },
    "category": { "type": "string", "enum": ["billing", "bug", "feature_request", "other"] },
    "priority": { "type": "integer", "minimum": 1, "maximum": 4 },
    "summary": { "type": "string" },
    "order_id": { "type": "string", "pattern": "^ORD-[0-9]+$" }
  },
  "required": ["reasoning", "category", "priority", "summary"],
  "additionalProperties": false
}

It is valid JSON Schema, and a useful test of the table above. llama.cpp accepts it. OpenAI’s strict mode rejects it because order_id is not in required. Anthropic’s rejects it because of minimum and maximum. The usual fix is to move constraints the provider cannot enforce into the field description and enforce them in your own validator, which runs on every output anyway:

import json
from jsonschema import Draft202012Validator  # one example of a validator

validator = Draft202012Validator(TICKET_SCHEMA)

def extract_ticket(text, call_model, max_attempts=2):
    messages = [{"role": "user", "content": PROMPT + text}]
    for _ in range(max_attempts):
        reply = call_model(messages)  # constrained or not, validate the same way
        if reply.stop_reason in ("refusal", "max_tokens"):
            return {"error": reply.stop_reason}  # not a parse error: do not retry blindly
        try:
            data = json.loads(reply.text)
            errors = [e.message for e in validator.iter_errors(data)]
        except json.JSONDecodeError as e:
            errors = [f"invalid JSON: {e}"]
        if not errors and passes_business_rules(data):  # e.g. the order exists
            return data
        messages += [{"role": "assistant", "content": reply.text},
                     {"role": "user", "content": "Fix: " + "; ".join(errors or ["order not found"])}]
    return {"error": "validation_failed"}  # fail visibly, never pass it on

• Check the stop reason first. A refusal or a cut-off reply is not a parse error to retry blindly.
• Validate against the full schema, including the constraints the provider could not enforce, then against business rules: an order_id that matches the pattern may still not exist.
• Retry with the validation errors in the message, and cap the attempts. Tam et al. also report that a second, corrective prompt reduced parse errors.
• When retries run out, fail visibly: return a typed error, queue the item for review or fall back to a simpler schema. Never pass unvalidated output on.
• Count the failures. Schema-failure and refusal rates are early warnings when a model or prompt changes; see LLM observability.

If you are building a feature that depends on reliable structured output, BYD Tech builds AI agents: the code lives in your GitHub from day one, and AI-assisted work is reviewed by an engineer.