How to verify LLM output
A language model writes a figure, a rule, a query or a block of JSON, and something is about to act on it. This guide covers how to check that output first: what each kind of check catches, which claims can actually be proven, and what to do with the ones that cannot.
The short answer
Verify the parts of the output that can be checked mechanically: numbers, logic, SQL, code and structure. Work them out again with a solver or parser instead of asking another model, let only what is proven go ahead, and treat everything else as unverified.
Three ways to check LLM output
Most teams combine these. They answer different questions, so one does not replace another.
| Method | Catches | Misses | Use it for |
|---|---|---|---|
| Ask another model | Tone, relevance, whether an answer looks off-topic or unsafe | Errors both models share; it is a judgement, so it can be confidently wrong | Content quality, where there is no formal answer |
| Check the shape | Missing fields, wrong types, malformed JSON (JSON Schema, Pydantic, regular expressions) | Whether the values inside the shape are true | Every structured output, as a first gate |
| Verify the claim | A sum, a logic condition, a SQL statement or a code pattern that is wrong, by working it out again | Anything that cannot be put into a formal statement, and what the user meant | Output that a system will act on |
The difference that matters is the last column. Asking a model whether another model was right gives you a second opinion. Working the claim out again gives you a result you can reproduce, which is what you want before money moves or a query runs. How the main tools differ
What can be verified, and how
Deterministic verification works on claims that can be written as a formal statement. Here is what QWED's engines cover:
| Kind of claim | Checked by | What comes back |
|---|---|---|
| Arithmetic, money and formulas | Math engine (SymPy) | VERIFIED when the result is proven |
| Conditions, rules and constraints | Logic engine (Z3) | VERIFIED only when solved; a solver timeout is never VERIFIED |
| SQL a model wrote | SQL engine (SQLGlot) | Destructive statements and injection patterns are denied |
| Code a model wrote | Code engine (AST and rules) | Unsafe patterns such as eval on input are denied |
| JSON and structured data | Schema engine (JSON Schema) | Shape checked against your schema |
| Facts stated in prose | Fact engine (advisory) | Signals for review, not a proof; held as UNVERIFIABLE |
Anything no engine can prove comes back UNVERIFIABLE and is denied by default. That is the point: a claim is never reported as verified because a check could not run. What fail-closed means
Step by step
- Decide what the output will drive. A summary a person reads needs different care from a refund, a query or a configuration change. Verify what a system acts on.
- Get the claim into a checkable form. Ask the model for structured output, or let it translate the text into a formal statement. Treat that translation as untrusted: it is not verified, so keep the formal statement with the result where a person can see it. Untrusted translator
- Check the shape first. Validate the JSON against a schema so the fields you need are there.
- Verify the claim. Recompute the figure, solve the condition, parse the query, inspect the code.
- Act on the decision, not the verdict alone. Let only
VERIFIED · ADMITgo ahead. HoldUNVERIFIABLEfor a person or another attempt; stopBLOCKEDandVERIFIED · DENY. Your application enforces this. Admission - Keep the record. Each verified result carries a
proof_ref, so a reviewer can later tell exactly what was checked.
Why LLMs get maths wrong, and how to check it
A language model produces the next likely piece of text. When it writes a calculation, the number is generated, not computed, so a long multiplication or a compound-interest figure can look right and be wrong. Giving the model a calculator tool helps, but the result it reports is still text you have to trust.
The reliable fix is to recompute the claim outside the model. QWED's Math engine uses SymPy to work the expression out again; financial formulas such as compound interest and NPV have their own checks in QWED Finance.
Checking structure: JSON Schema and Pydantic
JSON Schema, Pydantic models and the structured output modes of model APIs make sure the output has the right shape: the fields exist and have the right types. That is worth doing on every response. It does not tell you whether the values are true: a well-formed object can still hold a wrong total. Shape first, then verify the values that matter.
Try it
Install the core package and run the built-in checks. qwed test runs twelve deterministic checks and needs no model and no server:
pip install qwedqwed initqwed testWith the server running, verify a claim from Python. This is the example from the README:
from qwed_sdk import QWEDClientclient = QWEDClient(api_key="...", base_url="http://localhost:8000")result = client.verify_math("2*(3+4) = 14")print(result.status)Or over HTTP:
curl -X POST http://localhost:8000/verify/math \ -H "Content-Type: application/json" \ -H "x-api-key: $QWED_API_KEY" \ -d '{"expression": "2*(3+4) = 14"}'For SQL in particular, see Checking AI-generated SQL before it runs.
What this does not do
- It does not check intent. A correct answer to the wrong question is still correct.
- It does not prove facts in prose. "The contract was signed in 2019" has no formal proof; it stays unverified.
- It does not verify the translation. When a model turns text into a formal statement, that step is not itself checked.
- It is only as sound as its engines. Claims with a known answer run through the engines at every release, and the results are published on Evidence.