Validating LLM output with JSON Schema and Pydantic
Most applications ask a model for JSON so that code can use the answer. This guide covers how to validate that JSON, why a structured output mode does not settle it, and how to check that the values inside a well-formed object are right.
The short answer
Validate every structured reply against a JSON Schema or a Pydantic model before your code uses it, and reject anything that does not conform. Then check the values that matter, such as totals and rates, because a reply can match the schema exactly and still be wrong.
Structured output modes help, but do not finish the job
Model APIs can constrain a reply to a JSON Schema, and frameworks such as LangChain and the OpenAI SDKs can parse it straight into a Pydantic model. That removes most broken JSON and missing fields. Each layer catches something different:
| Layer | Catches | Misses |
|---|---|---|
| Structured output mode of the model API | The reply parses as JSON and follows the schema you sent | Whether any value is right |
| JSON Schema or Pydantic in your code | Types, required fields, ranges, patterns, enums, extra fields | Whether the numbers agree with each other |
| Value checks | A total that equals its parts, a tax that matches its rate, a figure that matches its formula | Whether the input data was right in the first place |
Keep a schema check in your own code even when the API enforces one: it is the same check every time, whichever model or provider produced the reply.
Shape is not truth
A schema says a field called total must be a number. It cannot say the number is right. An invoice of 100.00 plus 8.25 tax with a total of 118.25 passes every type check. In code that acts on the reply, that is the error that matters.
Checking shape and values with QWED
QWED's Schema engine (SchemaVerifier in the core package, pip install qwed) does both in one pass. It checks types, required fields, ranges, patterns, enums, nested objects and arrays, and rejects extra fields by default. For fields named like totals and tax, it also recomputes them with exact decimals: total = subtotal + tax - discount and tax = subtotal × tax_rate.
from qwed_new.core.schema_verifier import SchemaVerifierfrom qwed_new.core.diagnostics import admission_decisionschema = { "type": "object", "required": ["subtotal", "tax", "total"], "properties": { "subtotal": {"type": "number", "minimum": 0}, "tax": {"type": "number", "minimum": 0}, "total": {"type": "number"}, },}# What the model returned: well-formed, but the total is wrong.output = {"subtotal": 100.00, "tax": 8.25, "total": 118.25}result = SchemaVerifier().verify(output, schema)print(result.status.value, admission_decision(result).value)# VERIFIED BLOCKEDprint(result.developer_fields["issues"][0]["message"])# Total mismatch: expected 108.25, got 118.25BLOCKED in the second value is the admission decision; in a Verification Context record it is written DENY. For agents, the same checks are available as SchemaGuard in QWED Open Responses.
What comes back
| Reply | Result | Why |
|---|---|---|
{"subtotal": 100.00, "tax": 8.25, "total": 108.25} | VERIFIED · ADMIT | Matches the schema, and the total equals subtotal plus tax. |
{"subtotal": 100.00, "tax": 8.25, "total": 118.25} | VERIFIED · DENY | Matches the schema, but the total is wrong. The finding gives the expected value. |
{"subtotal": 100.00, "tax": 8.25} | VERIFIED · DENY | A required field is missing. |
A schema that is not a valid schema | BLOCKED · DENY | Nothing can be checked against it. |
VERIFIED means the check ran to completion; the decision says whether the reply passed. Use the reply only on VERIFIED · ADMIT, and send the findings back to the model so it can try again. How to verify LLM output
Pydantic or JSON Schema?
Both describe the same thing. Pydantic suits Python code that wants typed objects; JSON Schema suits checks that cross languages or services, and every Pydantic model can export one with model_json_schema(). Either way, the schema is the contract, and the check belongs at the point where the reply enters your system.
What it does not check
- Whether the inputs were right. If the model invented the subtotal, a total built from it can still add up.
- Values without a rule. Only fields with a known relation, such as total and tax, are recomputed. Other figures need the Math engine or a domain check.
- Facts in text fields. A string can be valid and false.