Structured Outputs: Valid JSON Is Not a Valid Result

Structured output makes an answer easier to consume. It does not, by itself, establish that the answer is true or that an application should act on it. Check four things separately: whether generation completed, whether the record meets your contract, whether its values are supported, and whether the requested operation is permitted.

OpenAI explicitly notes that structured responses can still contain mistakes. Google’s guide likewise calls for handling schema-compliant but semantically incorrect output. Those are distinct from a JSON parser rejecting malformed text. OpenAI guidance · Gemini guidance

This guide is for engineers moving an extraction or classification feature into a downstream workflow. It compares documentation checked on 29 October 2026, not model accuracy. The local example below tests a validator boundary, not any provider’s generation guarantee.

Keep four checks in the consumer

Original conceptual diagram. The checks are separate responsibilities, not measured stages or a guarantee that every error will be detected.

Check	Question	Example failure	Appropriate response
Completion	Did the API finish the intended result?	Refusal, output limit, transport failure	Use the documented non-success path; do not parse a partial result as final.
Contract	Does the complete record meet the application’s schema?	Missing key, unsupported enum spelling, wrong type	Reject or use a bounded repair path; retain the original validation error.
Meaning	Do values agree with the source and each other?	Invented date, wrong entity, inconsistent total	Recheck evidence or ask for review; valid structure is insufficient.
Permission	May this caller perform this operation on this object now?	Correct record, wrong account or stale approval	Enforce application authorization independently of model text.

This is our proposed consumer design, not a vendor-prescribed architecture. An extraction-only feature may end after the meaning check. A workflow that writes records needs its existing access controls and transaction safeguards as well.

For example, a model’s approved: true field can represent a prediction or an extracted statement. It should not become the source of the application’s authority to approve an operation. The consumer needs to know who authorized what, against which current record.

The schema you write may not be the one enforced during generation

Three contracts can exist: the schema in your source code, the schema sent over the wire, and the schema your consumer validates. Record all three when debugging a migration.

Documentation snapshot	Detail worth checking	Migration implication
OpenAI	Strict mode supports a JSON Schema subset; unsupported schemas cause errors. Required fields and closed objects are part of that contract.	A schema that works with a general validator is not automatically an accepted API request.
Claude Messages API	SDK helpers may remove unsupported constraints and place them in descriptions. A helper that validates responses checks the original constraints afterward.	Read the helper’s behavior. A description is guidance, not an equivalent decoder-enforced constraint.
Gemini API	The guide says some keywords, including string-length and uniqueness constraints, are stripped and ignored by the server; client-side validation is needed.	An accepted request does not prove that every submitted keyword constrained generation.

Sources: OpenAI supported schemas, Claude SDK transformation, Gemini limitations. These details are platform-specific, not a universal compatibility matrix. Hosted variants and SDK versions require their own check. Google’s page lists 21 October 2026 as its update date; the other two pages did not expose a clear revision date in the inspected text.

Claude’s guide also documents normally completed outputs whose string enum or const capitalization differs from the schema. For case-sensitive application values, keep a deliberate validation and mapping policy instead of silently accepting every spelling. Claude invalid-output cases

Do not loosen your application contract merely to remove validation errors. First decide whether the rule expresses a real requirement. If it does, preserve it at the layer that can enforce it.

A record can pass the schema and still be wrong

Consider this fictional source document:

“Order A-17. Currency USD. Subtotal 1000 cents. Tax 100 cents. Total 1100 cents.”

The deliberately small schema requires an order identifier, the currency USD, and three integer amounts. It forbids extra keys. General JSON Schema distinguishes a property’s type from whether the property is required; closing an object is another separate constraint. Object reference

{
  "type": "object",
  "properties": {
    "order_id": {"type": "string"},
    "currency": {"type": "string", "enum": ["USD"]},
    "subtotal_cents": {"type": "integer"},
    "tax_cents": {"type": "integer"},
    "total_cents": {"type": "integer"}
  },
  "required": ["order_id", "currency", "subtotal_cents", "tax_cents", "total_cents"],
  "additionalProperties": false
}

We checked three hand-written records locally with Python’s jsonschema 4.26.0 Draft 2020-12 validator. Each uses the keys above. None was generated by a model.

Synthetic record	Schema valid?	Subtotal + tax = total?	Matches the stated source?
A-17, USD, 1000 / 100 / 1100 cents	Yes	Yes	Yes
A-17, USD, 1000 / 100 / 900 cents	Yes	No	No
A-18, USD, 1000 / 100 / 1100 cents	Yes	Yes	No

The second record exposes a cross-field inconsistency. The third is internally consistent but names the wrong order. Adding a numeric lower bound would not fix that wrong identifier. A richer schema can encode more rules, but it still needs relevant evidence or application state to determine which real-world values are correct.

The source-match check here is exact equality against a hand-written expected record. It does not demonstrate a general document verifier, OCR accuracy, tax calculation, or model failure rate. Real documents can contain discounts, rounding, multiple currencies, or ambiguous identifiers. Define the correct invariant for that document type rather than imposing this example’s arithmetic everywhere.

Give missing evidence a valid destination

An extraction contract that requires an answer for every input can make uncertainty difficult to represent. OpenAI warns that unrelated input can lead to hallucinated values and recommends specifying how incompatible input should be handled. Input-handling guidance

For your own domain, consider an explicit result state such as extracted, insufficient_evidence, or not_applicable, with appropriately nullable fields. Validate the relationship between that state and its fields in application code when the API’s schema subset cannot express it.

A source reference helps a reviewer locate evidence; it does not prove that the referenced text supports the claim. Preserve document identifiers and versions, and check the relevant passage rather than trusting a model-generated citation string. If a source is absent or ambiguous, route the case to the chosen fallback instead of converting uncertainty into a fabricated value.

Count useful results, not just parsable responses

For a fixed evaluation set, record these separately:

• Attempts: all requests, including retries and non-success outcomes.
• Completed records: responses that finished without a refusal or truncation condition.
• Contract-valid records: complete records accepted by the application validator.
• Accepted task results: records that also meet the task’s evidence and correctness criteria.
• Cost per accepted task: the total cost of all attempts divided by accepted tasks, if any.

Keep an initial-task denominator as well as an attempt denominator: retries can improve eventual success while increasing cost and delay. Include missing-field, irrelevant-input, wrong-entity, conflicting-source, and partial-output cases. Label synthetic cases separately from representative production data, and do not turn a tiny test suite into a reliability percentage.

For retry economics, see prompt-cache cost accounting. For workflows that turn model output into repository changes, safe-output review covers the additional execution boundary.

Before shipping: capture the exact model/API/SDK and schema version, inspect the transmitted schema, test the consumer’s non-success paths, and test that known invalid cases are rejected or routed for review before consequential actions. Structured output is a useful contract boundary; use it alongside the other checks your application actually needs.