The generator is allowed to be wrong
There was a stretch this week where every third thing in my feeds was a language model doing real mathematics. Terence Tao working through a candidate counterexample to the Jacobian Conjecture in a ChatGPT window. A claim that Claude had produced a counterexample to the same conjecture from a cold prompt. A note that GPT-5.6 had been talked into closing a thirty-year gap in a convex-optimisation proof. Whatever survives the referees by the time you read this, the shape was unmistakable: the models are now generating mathematics that working mathematicians take seriously enough to check.
And enough to check is the phrase I keep turning over, because it is quietly doing all the work.
My first reaction to any of this is the engineer’s reaction, and it’s a flinch. I have spent a career learning to distrust components that are wrong some fraction of the time. A function that returns the right answer 95% of the time is not a function I want three levels down in a call stack; it’s a bug I haven’t finished fixing. So when someone tells me that a system which hallucinates, which will assert falsehoods with total composure, which cannot even tell you its own error rate, has just produced a counterexample to a famous conjecture, the flinch says: you cannot build on that. It’s unreliable by construction. How could you trust the output?
The answer, in the case of the mathematics, is wonderfully blunt. Nobody trusted it.
Nobody trusted the model
A counterexample is the easy case, epistemically. You are handed a specific object — a polynomial map, say — and asked whether it does the thing. So you plug it in. You compute the Jacobian, you check whether it’s a nonzero constant, you see whether the map fails to be invertible after all. The verification is mechanical, cheap and exact, and it does not care in the slightest how the candidate was produced. A mathematician who wouldn’t take the model’s word for the time of day will happily take its counterexample, because the counterexample doesn’t require taking its word for anything. The model does the hard, creative, unreliable part — proposing the object — and a check a competent undergraduate could run does the deciding.
The unreliability of the generator is real, and it simply doesn’t matter, because the generator never gets a vote on whether the answer is right. That vote belongs entirely to the check.
Once I saw it in the mathematics I couldn’t stop seeing it, because it’s the shape of the thing I do all day on the other side of my working life. Engineers know the abstract version — it’s the whole character of NP, that finding a solution can be brutally hard while checking one is easy — but we rarely reach for it as a tactic. Data scientists reach for it constantly, and give it no name because it’s simply how the work is arranged. You never trust the model. You trust the held-out set. A model is a machine for producing plausible answers; the evaluation is the apparatus that decides whether to believe them, and the entire discipline is built so that the believing happens at the checker and never at the generator. Train however wildly you like, on whatever hunch — the test set doesn’t ask where the candidate came from, only whether it holds up. The model proposes; the held-out set disposes.
Mathematics has a free oracle
Here is the part the headlines quietly bury. The maths results feel safe not because the models crossed some reliability threshold this month, but because mathematics is the luckiest possible domain: correctness is decidable, and the decision is cheap. A formal proof either type-checks or it doesn’t. A counterexample either is one or it isn’t. There is an oracle, it’s nearly free to consult, and it is never in a good mood or a bad one. That is a luxury, and it is rare.
You feel how rare the moment you step an inch outside it. This same week Simon Willison shipped a small tool that annotates SQLite query plans, and he was honest about the catch: he doesn’t know enough about SQLite’s internals to verify the explanations himself, so he shrugged that they “seem cromulent enough” to him. That shrug is the whole distance between the two worlds. In mathematics there is no seems; there is a check. In most of software there is only seems, because the oracle is expensive, or ambiguous, or absent — and “seems cromulent enough” is precisely the verdict you’re left with when you have a confident generator and no cheap way to call its bluff.
Most software hasn’t got one
This is where the engineer has to be careful, because the maths headlines flatter a wrong conclusion. They make it look as though the models became trustworthy, when what actually happened is that a domain handed them a cheap verifier. Move the same model to a place where correctness is a matter of taste, or of downstream side-effects, or of whether the customer meant what you assumed they meant, and its being-wrong-sometimes stops being harmless in the same instant — because now nothing catches it. The output that looks right and the output that is right are indistinguishable without a check, and looks right is exactly what a plausible-answer machine is optimised to produce. A hallucination and a theorem are the same shape right up until something evaluates them.
So the useful question stops being “is the model reliable enough yet?” — a question about the generator — and becomes one I only learned to ask from the other side of the fence: what is my verifier, how cheap is it, and how much of the correctness can it actually certify? Sometimes the verifier is a test suite. Sometimes it’s a type system, a schema, a property-based check, a constraint the output has to satisfy, a compiler, a second and cheaper model whose only job is to say no. The work of putting a probabilistic model into production was never really about making the model trustworthy. It was about building the deterministic frame around it that is allowed to reject what it produces. That’s most of what Building with Certainty is about, when you get past the mechanics: how you take the happy-path model out of a notebook and wrap it in the validation, the tests, the acceptance checks that turn an unreliable component into a reliable system — not by fixing the generator, but by refusing to trust it.
Let it be wrong
The generate-and-verify framing even explains the pattern of where the models are winning. They advance fastest exactly where the checking is cheapest — formalised proofs, small self-contained counterexamples, competition problems with a known answer, code with a test that either goes green or doesn’t — and they stall where verification is hard or slow or a matter of judgement. The bottleneck was never really generation. It was whether you can afford to check. Give a model a domain with a free oracle and you can tolerate any quantity of nonsense on the way to the answer, because the nonsense gets filtered at the door. Take the oracle away and that same nonsense is no longer a discarded draft — it’s your output.
Which is why I’ve made my peace with the flinch. The engineer in me wants a component that is right. The data scientist in me has learned to want something more achievable and, honestly, sturdier: a component that is allowed to be wrong, sitting behind something that is allowed to say no. Reliability doesn’t have to live in the generator, and it’s cheaper and more robust when it doesn’t. Build the checker, make it cheap, give it the final vote — and then let the generator be as wrong as it likes, right up until the moment something checks.
Part of an occasional series reframing everyday engineering through a data scientist’s eyes. The ideas here are developed properly in Thinking in Uncertainty and Building with Certainty.