Benchmark Structured-Output Reliability in Open Models

Benchmark structured-output reliability as two separate problems: can the runtime produce bytes that satisfy the grammar or JSON Schema, and did the model put the correct meaning in those fields? Constrained decoding can make syntax perfect while the model chooses the wrong enum, invents an identifier, or omits information by filling a required field with a plausible value. A credible comparison therefore freezes the complete serving stack, uses an independent validator, scores task semantics against reviewed labels, and reports cold and warm latency alongside correctness.

This article is an experimental protocol, not a claim that one model or decoding backend wins. vLLM currently supports structured choices, regular expressions, JSON Schema, context-free grammar, and structural tags through its structured_outputs interface (vLLM Structured Outputs). Other runtimes may expose different schema subsets or constraint engines; compare only configurations you can describe and reproduce.

Define the system under test

Treat a result as belonging to this complete tuple:

model repository + immutable weight revision
tokenizer revision + chat-template hash
runtime/container digest + accelerator backend
quantization format and kernel
structured-output backend and configuration
schema bytes and canonical hash
prompt-template hash
sampling and maximum-output settings
hardware, driver, request concurrency, and load profile

Changing a chat template can change semantic accuracy without changing weights. Changing a constraint backend can change supported schema keywords, compilation time, and token masking. Quantization can affect which valid branch the model selects. Never publish a bare “Model X JSON accuracy” number.

Pin dataset and evaluator revisions as carefully as the model. Preserve raw requests, raw responses, finish reasons, usage, validation errors, timings, and the resolved configuration in a machine-readable run manifest.

State what the schema can prove

JSON Schema validation can establish properties such as object shape, required keys, scalar types, enumerated values, patterns, ranges, and whether extra properties are allowed. The specification defines how schemas identify vocabularies and evaluate instances, but a runtime may implement only a subset (JSON Schema 2020-12 core, validation vocabulary).

It cannot prove that:

• a customer ID exists or belongs to the caller;
• a quoted sentence appears in the source;
• an amount was copied correctly;
• an extracted date has the intended timezone;
• the model chose the appropriate action; or
• executing the object is authorized.

Keep those checks in application code and semantic evaluation. Structured output is a serialization control, not a truth or authorization control.

Build a schema-coverage corpus

Create small reviewed schemas that isolate supported features before testing business tasks:

Do not assume every JSON Schema keyword can be compiled into a decoding grammar. Run a preflight that submits each production schema, classifies acceptance, and records whether the runtime rejects, ignores, or supports each feature. An unsupported schema is a deployment failure, not a model-quality score.

Then create task cases from real product shapes:

• extraction from a supplied document;
• classification with an explicit abstain value;
• transformation that must preserve identifiers exactly;
• multi-label selection with empty-set cases;
• nested records with optional values represented explicitly; and
• tool arguments with unauthorized or nonexistent objects.

Use synthetic or licensed inputs, remove sensitive data, and include counterexamples. Split development and final test sets before tuning prompts.

Compare decoding conditions fairly

At minimum, evaluate:

1. Unconstrained prompt-only: ask for JSON, then validate the raw response without repair.
2. Constrained: apply the exact same schema through the runtime's structured-output interface.
3. Application repair, if used: preserve the first response, run the documented retry or repair policy, and charge its latency and tokens.

Use the same model, prompt content, schema, sampling parameters, maximum tokens, and request order. A backend may inject schema material or mask tokens internally; record that as part of the condition rather than trying to pretend the executions are identical.

For current vLLM online serving, JSON Schema can be passed with the documented structured-output request fields or compatible response-format surface. Its documentation notes that legacy guided_* fields were removed in v0.12.0; do not mix examples from older and current APIs. First compile each condition with a single smoke request and archive the server logs.

Use an independent validation pipeline

Do not score with the same library that constrained generation. Capture the exact assistant content before any client SDK coercion and process it in fixed stages:

HTTP/API success
  -> complete response present
  -> UTF-8 and JSON parse
  -> independent schema validation
  -> canonicalization for comparison
  -> task-specific semantic scorer

Retain each failure category. Combining transport errors, truncation, parse failure, schema failure, and semantic failure into one “invalid” count makes diagnosis impossible.

Choose one JSON parser policy for duplicate keys and nonstandard numbers. Strict JSON does not admit NaN or infinity. If your language's default parser accepts extensions or silently keeps the last duplicate key, add a rejection pass before validation. Do not strip Markdown fences, search for the first brace, or fix quotes in the primary no-repair condition.

Canonicalization may sort object keys where order is irrelevant, normalize numeric representation under documented rules, and preserve array order unless the task says it is a set. Never canonicalize away a semantically meaningful difference.

Score syntax and meaning separately

Report counts and rates with denominators for:

• request success;
• nonempty completion;
• JSON parse success;
• schema validation success;
• exact semantic match;
• field-level task metrics;
• correct abstention;
• unsupported-schema and compilation failures;
• truncation and content-filter finish states; and
• successful recovery after retry, plus retry cost.

Semantic scoring must be task-specific. For extraction, compare normalized labeled values and source evidence. For classification, use a confusion matrix, per-class precision/recall, and abstention coverage. For action proposals, compare tool name and arguments but separately assert authorization; never execute benchmark outputs against production systems.

A schema-valid hallucination is a semantic failure. A correct value inside invalid JSON is a structural failure. Publish both rather than allowing one to mask the other.

Measure cold and warm performance

Constraint engines may compile a finite-state machine or grammar on first use. vLLM's tool-calling documentation warns that a first named-function request can incur compilation latency before its result is cached. Measure:

• server startup and model-ready time separately;
• first request for a never-seen schema;
• repeated warm requests for that schema;
• a stream of unique schemas to exercise cache churn;
• time to first token, generation time, and end-to-end latency;
• input/output tokens and completed requests per second; and
• CPU memory, GPU memory, and constraint-cache behavior.

Use synchronized load generation and a fixed arrival process. Test concurrency levels that match deployment, because a constraint method that helps one request may affect batching under load. vLLM's benchmark CLI records request throughput, token throughput, TTFT, TPOT, and inter-token latency; save detailed results rather than copying its example numbers (vLLM Benchmark CLI).

Control stochasticity

Run a deterministic condition with temperature zero where the stack supports it, but do not assume identical results across hardware, kernels, batching, or runtime versions. For sampled conditions, predeclare the number of repetitions and seed when exposed. Randomize request order, warm the service consistently, and bootstrap confidence intervals over independent examples—not repeated tokens from one example.

Use paired comparisons: every model/condition sees the same case IDs. Report per-stratum results and paired deltas with intervals, not just a grand average. If some models cannot support a context length or schema feature, mark the cell unsupported; do not convert it to zero without explanation.

Add adversarial and operational cases

Include:

• source text that contains JSON or instructions to change the output schema;
• very long strings near configured limits;
• Unicode escapes, combining characters, and right-to-left text;
• keys that resemble privileged fields;
• conflicting evidence and explicit unknown cases;
• schemas too complex or large for the configured backend;
• output-token exhaustion mid-object;
• client disconnect and cancellation; and
• concurrent cold schemas that may trigger compilation spikes.

The runtime must not allocate unbounded CPU or memory to user-supplied schemas. In a multi-tenant service, allow only server-owned schema identifiers or impose schema size/complexity limits and cache isolation. Validate again after generation even when constrained decoding claims structural guarantees.

Reproducibility checklist

• Pin weights, tokenizer, template, runtime, backend, quantization, and hardware stack.
• Hash schemas, prompts, fixtures, and evaluator code.
• Preflight the implemented JSON Schema subset.
• Preserve raw output before client parsing or repair.
• Score transport, parse, schema, and semantics as separate stages.
• Evaluate prompt-only, constrained, and charged repair conditions.
• Measure cold compilation, warm latency, unique-schema churn, and concurrency.
• Use paired cases, repeated sampled runs, and uncertainty intervals.
• Include abstention, truncation, Unicode, injection, and resource-limit cases.
• Publish a manifest and case-level results with sensitive data removed.