Getting Reliable Structured Outputs from LLMs: Validation, Retries and Constrained Decoding

The Evolution of Structured Output Requirements

The industry standard for data extraction has shifted rapidly over the past 24 months. Early implementations relied heavily on prompt engineering, where developers would instruct models to format their responses according to strict JSON structures. However, these methods were prone to "hallucinated" keys, malformed syntax, and inconsistent data types.

The current landscape, as of late 2026, features native support from major providers. OpenAI’s Responses API, for instance, now allows developers to define a Pydantic model or a specific JSON schema that the model is architecturally forced to follow. Similarly, open-source frameworks like vLLM have integrated tools such as xgrammar to enforce grammar constraints at the tokenization layer. Despite these technological leaps, many engineering teams mistakenly believe that the successful parsing of a JSON string marks the end of the validation process. This oversight is a frequent source of "silent failures," where the model produces a syntactically correct output that is semantically inaccurate or fundamentally disconnected from the source material.

The Reality of Semantic Inconsistency

To illustrate the danger of relying solely on schema validation, consider the processing of a standard corporate invoice. If a system is tasked with extracting an invoice_id, currency, and total_cents from a document stating a total of 1,250.00 EUR, a high-performing model might generate a JSON object containing total_cents: 12500.

While this object is perfectly valid JSON and satisfies the schema requirements—it is an integer, it is correctly formatted, and it matches the requested keys—it is mathematically incorrect. The value 12,500 cents equates to 125.00 EUR, a discrepancy of 1,125.00 EUR from the source document. A standard parser will accept this as valid data because, from a technical perspective, there is no structural flaw. This highlights a fundamental divide: syntactic validation (does the JSON parse?) versus semantic validation (is the information factually supported by the source?).

A Three-Tiered Defense Strategy

To mitigate these risks, architects are moving toward a multi-layered validation framework comprising three distinct phases:

1. Contract-Based Validation: Using strict schema definitions (e.g., Pydantic 2.14+) with features like strict=True and extra="forbid". This ensures the model does not attempt to "coerce" data types—such as turning a string into an integer—or inject unauthorized metadata into the output.
2. Source-Text Verification: This is the application-level logic that compares the extracted values against the raw source material. If a model extracts a total_quote, the system must verify that this string exists verbatim in the original document. If the extracted numerical value does not mathematically align with the quoted line, the record must be rejected.
3. Constrained Decoding: This is the most advanced layer, where the model’s generation process is restricted at the token level. By masking out tokens that would violate the defined schema, the system prevents the model from ever generating a "shape error" (such as a missing bracket or a forbidden character) before it even reaches the application layer.

The Logic of Controlled Retries

A common pitfall in automated pipeline design is the "infinite retry loop." Naive implementations often attempt to re-prompt the model indefinitely upon any failure. This is inefficient, costly, and often ineffective.

Instead, professional pipelines categorize failures into terminal and transient classes. A "shape error"—where the output is not valid JSON—is a candidate for a limited retry, as the model may simply have had a minor tokenization misstep. However, if the output is structurally correct but semantically wrong (e.g., the wrong total), a retry is often useless, as the model is likely to repeat the same error based on its interpretation of the prompt. In such cases, the system should trigger a "needs_review" status, routing the document to a human analyst rather than risking a downstream write of incorrect data.

Chronology of Pipeline Execution

1. Ingestion: Raw text is extracted from the document (e.g., via OCR).
2. Constraint: The model is invoked with a constrained decoding strategy (e.g., client.responses.parse).
3. Primary Validation: The response is checked for completion. If the model refuses or provides an empty result, the process halts.
4. Semantic Check: The parsed Pydantic object is cross-referenced with the raw source text.
5. Resolution: If valid, the data is committed. If invalid, the system determines whether the error is recoverable through a controlled retry (limited to three attempts) or requires manual intervention.

Broader Implications for Enterprise AI

The shift toward rigorous validation reflects a broader maturation of the AI industry. As companies like Northwind Traders and others scale their use of LLMs, the focus has moved from "Can it do the task?" to "How do we guarantee the outcome?"

Financial audits and regulatory compliance requirements demand that automated systems maintain a clear audit trail. By implementing rigid contracts, developers provide a layer of accountability that is easily testable. If a pipeline fails, the logs can pinpoint exactly which constraint was violated: was it a schema mismatch, a discrepancy with the source text, or an exhausted retry budget?

Furthermore, this approach acknowledges the inherent limitations of models. No matter how advanced a model becomes, it operates on probabilistic patterns rather than absolute truth. By treating the model as a participant in a broader, deterministic application flow, businesses can capture the efficiency of generative AI without sacrificing the precision required for high-stakes operations.

Future Outlook

As we move into 2027, the reliance on these validation layers will likely become the baseline for all enterprise-grade AI. The development of more robust, native schema-enforcement tools within LLM providers will continue to reduce the frequency of low-level errors, but the necessity for application-level semantic verification will remain. In a world where data accuracy is synonymous with business integrity, the ability to validate the output against the source will remain the most critical skill for the next generation of AI engineers. Ultimately, the goal is not to force the model to be perfect, but to build a system that is sufficiently skeptical of the model to ensure that only the most reliable data makes it into the permanent record.