Open-source runtime verification for AI, licensed under Apache-2.0.
QWED
Contact

Type to search pages, engines and advisories.

    Looking for API details?Search the docs

    Can LLMs do maths? How to check AI calculations

    Language models write fluent explanations and confident numbers, and the numbers are sometimes wrong. This guide covers why that happens, what a calculator tool does and does not fix, and how to check every figure before a payment, a report or an agent acts on it.

    The short answer

    LLMs can often get maths right, but nothing guarantees it: a model predicts text, it does not compute. When a figure matters, do not trust it or ask another model. Recompute the claim outside the model with a computer algebra system such as SymPy, and act only on what checks out.

    Why language models get maths wrong

    • They predict, they do not calculate. A model writes the most likely next piece of text. A result that looks right is likely text, and that is not the same as a correct result.
    • Numbers are split into pieces. Models read text as tokens, and a long number can be cut into chunks that do not line up with its digits, which makes carrying and place value harder.
    • Long chains compound. A calculation with many steps gives many chances for one small slip, and every later step builds on it.
    • The tone does not change. A wrong total is written with the same confidence as a right one, so a reader cannot tell them apart.

    Newer models, and models that write out their working, make fewer of these mistakes. None of them guarantees an exact answer.

    Is a calculator tool enough?

    Giving the model a calculator or a code tool helps a lot: the arithmetic itself is then done by a machine. Two gaps remain. The model still decides what to calculate, so it can compute the wrong formula correctly. And it still copies the result into its answer, where a figure can change on the way. The QWED MCP server is one such tool: it lets MCP clients run Python calculations in a restricted subprocess instead of guessing.

    The fix for both gaps is the same: check the claim in the final output, not just the steps the model took.

    How to check a calculation

    1. Get the claim as an expression. Ask the model for structured output, such as the formula and the figure it claims, or extract them from its answer.
    2. Recompute it. QWED's Math engine parses the expression with SymPy, with no general code evaluation, and checks it. An identity a = b holds when a - b simplifies to zero.
    3. Act on the result. Use the figure only when it comes back VERIFIED · ADMIT. Anything else goes back to the model, or to a person. Fail-closed, explained

    What comes back

    ExpressionDecisionWhy
    2*(3+4) = 14VERIFIED · ADMITThe two sides simplify to the same value.
    2*(3+4) = 15UNVERIFIABLE · DENYThe identity is false; the difference (-1) is in the findings.
    (x+1)**2 = x**2 + 2*x + 1VERIFIED · ADMITTrue for every x, shown symbolically, not by trying values.
    1/0BLOCKED · DENYDivision by zero: there is no value to check.
    6/2(1+2)BLOCKED · DENYAmbiguous: implicit multiplication after a division can be read two ways.

    The first word is the verdict, what the engine could prove; the second is admission, whether the figure may be used. In qwed 7.3.0 the Math engine returns the verdict, and the admission follows the rule in the Verification Context specification: only a VERIFIED result that is valid is admitted, and everything else is denied.

    A false identity comes back UNVERIFIABLE · DENY: the engine does not certify a wrong figure, and your application should not use it. On the main branch, not yet in a release, a false identity is reported as proven false, VERIFIED · DENY, and the response carries the admission itself. A VERIFIED result carries a proof_ref, so the check can be reviewed later. Audit trails for AI decisions

    Try it

    With the core package (pip install qwed) and the server running, this is the example from the README:

    python
    from qwed_sdk import QWEDClientclient = QWEDClient(api_key="...", base_url="http://localhost:8000")result = client.verify_math("2*(3+4) = 14")print(result.status)

    Or over HTTP, here with a figure that is off by one:

    shell
    curl -X POST http://localhost:8000/verify/math \  -H "Content-Type: application/json" \  -H "x-api-key: $QWED_API_KEY" \  -d '{"expression": "2*(3+4) = 15"}'

    Money and formulas

    Financial figures need more than arithmetic: the right formula, the right compounding and exact decimals. QWED Finance recomputes NPV, IRR, future and present value, loan payments, amortisation schedules and total interest, so a model's figure can be checked against the formula it claims to use. Banking and payments

    What it does not check

    • Whether it is the right calculation. The engine checks the expression it is given. If the model picked the wrong formula, the check proves the wrong formula is computed correctly.
    • Word problems on their own. When a model translates a question in plain language into an expression, the translation is not verified, so that path never returns VERIFIED · ADMIT. Send the expression itself to /verify/math. Untrusted translator
    • Everything SymPy cannot settle. Results are only as sound as SymPy and QWED's own rules. Math engine reference

    Everything QWED does not verify