Twenty percent faster is not a result
I said it to a colleague a couple of weeks ago, without really thinking about it: the new tooling had made me about twenty percent faster. No pushback. It’s the kind of number that slides straight through — round, confident, plausible — and I only caught that I’d said it because I read a post this week taking exactly that sort of claim apart. It was about AI and developer productivity, all the breathless figures about how much faster we’re all supposed to be now, and its patient question was: faster than what, measured how, and how would you know? I recognised the number it was dismantling. It was mine.
The thing is, I’d never let a number like that past me if it came out of a benchmark. That’s home ground. When I want to know whether a change made something faster, I don’t run it once and read off the result — one run tells you almost nothing. I run it many times, throw away the first few while the caches warm up, watch how much the figure moves between runs, and check that nothing else was competing for the machine while I did it. I’ve watched a change that looked twenty percent faster on the first run drift back to almost nothing by the tenth, on an idle box, measuring the same code against itself. The single number is never the thing. The spread around it is the thing, and a figure reported without it is a point estimate in the costume of a fact.
What took me embarrassingly long to see — years, really — is that all of that care is statistics. I thought I was just being a diligent engineer. I was computing a crude effect size and eyeballing its standard error, deciding whether a difference was bigger than the noise it was sitting in, and I’d have been faintly annoyed if you’d told me that’s what it was. When I finally read the formal version, the thing that unsettled me wasn’t that it was unfamiliar. It was how familiar it was — the same suspicion I’d been applying to a build time for years, written down properly, with the parts I’d been doing on instinct given their real names.
Faster than what
So here is what I now can’t help asking of “twenty percent faster.” Faster than what baseline? A benchmark result means nothing without the thing it’s measured against, and it has to be the same thing — same inputs, same machine, changing only the one variable you’re actually testing. The productivity figure almost never has that. Twenty percent faster than the same team last quarter — the quarter before they’d learned the codebase, on a different set of tickets, some of which were always going to be easier? That isn’t the same box running the same code. It’s two different situations with a dozen things moving at once, and the tool is only one of them. In a benchmark I’d throw that result out and start again. I’ve been much slower to throw out the productivity version, and I’ve come to think it’s because the benchmark number is one I’m neutral about and the productivity number is one I want.
There’s a second thing the single figure hides, which is everyone it’s averaged over. Say the twenty percent is real and cleanly measured. It’s still a mean, and a mean can be assembled from a team where half the people got meaningfully faster, a few got slightly slower, and one got badly stuck — and the average slides right over the person the tool is quietly costing you. I’d never report a benchmark as a bare mean across wildly different workloads without at least glancing at the distribution underneath. The productivity number is almost always the bare mean, and the distribution underneath it is where the interesting, inconvenient part of the story tends to live.
When the number becomes the target
Even a clean twenty percent wouldn’t quite settle it, because there’s a prior question the benchmark instinct doesn’t reach on its own: whether the thing I measured is the thing I care about. Velocity, pull requests merged, tickets closed, lines written — these are what you can count, so they’re what gets counted, and the moment one of them becomes the number you’re trying to move, it stops tracking whatever it was standing in for. You can ship twenty percent more pull requests and produce less that matters; the measure and the goal come apart precisely because you’re now optimising the measure. A statistician would call it a proxy problem. Engineers have been burned by it forever — we just usually meet it as the metric someone started gaming the week after it went up on a dashboard, rather than as anything with a name.
The number I don’t hold to the same standard
What I notice, doing all this, is that the rigour is easy to summon for other people’s numbers. Someone else says a tool made them forty percent more productive and the questions arrive on their own: forty percent against what, over how long, measured how, and would it have happened anyway. I’ve got quite good at that. What I’m still bad at is turning it on the numbers I’m pleased with. My own twenty percent went out in a standup without a single one of those questions attached, because I wanted it to be true, and a round, confident figure is a very comfortable thing to want. The instinct I trust completely at a benchmark harness deserts me at exactly the moment the result is flattering and mine. I haven’t fixed that. I’ve just started noticing, a beat too late, which of my numbers I’ve quietly let off.
Part of an occasional series reframing everyday engineering through a data scientist’s eyes. The ideas here are developed properly in Thinking in Uncertainty and Building with Certainty.