Articles

Short-form writing on topics that sit at the intersection of software engineering and data science. New articles appear here as they’re published.

A histogram of random-SSD-read latency in microseconds, strongly right-skewed with a long tail to the right. A solid blue vertical line marks the memorised textbook value at about 16 microseconds, sitting near the tall left peak. A dashed amber line marks the mean, a few microseconds to the right of it. A dotted green line marks the 99th percentile, far out along the tail past 60 microseconds. The bulk of the mass is near the textbook value, but a visible tail stretches well beyond it.

The numbers you memorised were a distribution

foundations
engineering

Every engineer carries the same little table of latency constants: memory is ~100 ns, an SSD read is ~16 µs. We quote them like physical constants. But each entry was only ever the centre of a wide, drifting distribution, measured on particular hardware in a particular year — a point estimate wearing the costume of a fact.

7 Aug 2026
Matthew Gibbons
 

Twenty percent faster is not a result

inference
engineering

A productivity headline reads like a benchmark result, but it’s a treatment effect with no baseline, no spread, and no control. The engineer who re-runs a benchmark ten times already owns the instinct the number is missing — he just doesn’t always turn it on numbers he wants to be true.

1 Aug 2026
Matthew Gibbons
 

The generator is allowed to be wrong

foundations
testing

An engineer meets an unreliable component with dread: how do you build on something that’s wrong some of the time? But mathematics accepts a model’s bad guesses happily, because a counterexample either satisfies the equations or it doesn’t. The trust was never in the generator — it was in the checker. And most of software’s trouble is that it hasn’t got a cheap one.

24 Jul 2026
Matthew Gibbons

A chart with days since release on the x-axis, from 0 to 14, and estimated serious-problem rate as a percentage on the y-axis, from 0 to about 50. A shaded blue band shows the 95% confidence interval on the true rate. At the far left the band is very tall, reaching from 0 up to roughly 45%. It narrows sharply over the first few days and by day 14 has collapsed to a thin band around 2%. A blue line with dots traces the rate the crowd has actually observed, starting at or near zero and settling onto 2%. A grey dotted horizontal line marks the true rate of 2%. A dashed orange vertical line at day 3 is labelled 'three-day cooldown', sitting where the band has just become narrow.

Give the release time to gather evidence

inference
engineering

Bumping to the newest release feels like hygiene: newer is better, and a green build is a green light. But a day-old release is the least-tested it will ever be — time since publication is sample size, and on day zero your sample is almost nobody. GitHub’s new dependency cooldown isn’t caution for its own sake; it’s waiting for the standard error to come down.

18 Jul 2026
Matthew Gibbons

A line chart with a logarithmic x-axis running from 2 to 100, labelled 'Number of options you optimised over'. The orange line, 'Report the raw winning score', rises steadily from about 0.4 at two options to about 1.8 at one hundred. The blue line, 'Report the shrunk estimate', stays flat along zero across the whole range. A dotted grey line marks zero. The y-axis is labelled 'Average overstatement of the winner (reported minus true)'.

The best result is chosen partly for luck

inference
engineering

Optimising is the engineer’s purest reflex: run the options, take the top. But the maximum of a set of noisy measurements is biased upward — you selected partly for quality and partly for luck — so the winner regresses and your best result disappoints, not by accident but on schedule. The cure isn’t more precision; it’s shrinking the number back towards what you expected before you looked.

3 Jul 2026
Matthew Gibbons
 

Your spreadsheet was always a database

engineering
collaboration

The industry has decided that non-developers building their own tools is a dangerous new thing. It isn’t. We have run businesses on Excel ‘databases’ and Word ‘forms’ for decades, and the only genuinely new entrant is the one I was shunned for myself: the data scientist’s notebook. The problem was never the tool someone reached for. It’s that nobody noticed the day it became integral.

26 Jun 2026
Matthew Gibbons

A curve falling steeply from left to right. The x-axis is the base rate of high-impact reports on a logarithmic scale, from 1-in-2 down to 1-in-500. The y-axis is positive predictive value as a percent. The blue curve starts near 90% at a base rate of 1-in-2, passes through about 51% at 1-in-10, about 16% at 1-in-50, and falls below 5% at 1-in-200. Three points are marked and annotated with the number of false alarms per genuine find: roughly 1 at 1-in-10, roughly 5 at 1-in-50, roughly 20 at 1-in-200.

Your security inbox is a screening test

inference
engineering

For years a vulnerability report was special: scarce insight, offered confidentially, owed a response. Now that anyone’s LLM can find the same bugs, the report isn’t the scarce thing — the triage is. And the moment volume outruns the base rate, triage stops being a queue of obligations and becomes a screening test, with all the cruel arithmetic that implies.

25 Jun 2026
Matthew Gibbons

Horizontal interval plot with two rows. The top row, 'Model A (1st)', shows a blue dot at 71.2% with a horizontal bar spanning roughly 66.8% to 75.6%. The bottom row, 'Model B (2nd)', shows an amber dot at 70.4% with a bar spanning roughly 65.9% to 74.9%. The two bars overlap across almost their entire length. A small bracket between the two dots is annotated 'leaderboard gap: 0.8 pts', and a caption notes the interval half-width is about ±4.4 points — far larger than the gap.

The leaderboard has no error bars

inference
engineering

A model leaderboard reads like a build number: one authoritative ranking, settled. But an eval score is a sample statistic, and the gap between first and second is usually well inside the noise. A ranking reported without an interval is a point estimate wearing the costume of a fact.

19 Jun 2026
Matthew Gibbons
 

The crash that didn’t happen

inference
engineering

Reliability work is invisible when it works. That’s not a management problem — it’s a causal one: the value of prevention lives in the branch of history you never got to observe.

12 Jun 2026
Matthew Gibbons
 

A stranger to your own codebase

engineering
collaboration

Agents hand you the code without the understanding that used to come with writing it, so you re-onboard onto your own project every iteration. In modelling you can ship what you can’t explain — but code isn’t a model, and someone has to keep understanding it.

5 Jun 2026
Matthew Gibbons
 

The test that couldn’t have failed

inference
testing

A green tick only means something if a broken version of the code would have turned it red. Pick the wrong input and you can write a test that passes no matter what — proof of nothing, in a reassuring shade of green.

22 May 2026
Matthew Gibbons

Line chart with consecutive passing runs (zero failures observed) on the x-axis from 0 to about 300, and plausible true failure rate as a percentage on the y-axis from 0 to 30. A blue curve labelled 'Exact 95% upper bound' falls steeply from the top left and flattens as it approaches the x-axis; an amber dashed curve labelled 'Rule of three (3/n)' tracks just above it. A red-orange dot at 20 runs sits at about 14%, annotated '20 green runs still allow a ~14% failure rate'. A dotted grey horizontal line at 1% is annotated to show that roughly 300 runs are needed before the bound drops below it.

Your flaky test is a hypothesis test

inference
testing

Quarantining a flaky test assumes its true pass rate is 100%. It isn’t — a test that touches anything real is a Bernoulli trial, and nineteen green runs prove far less than you’d think.

17 Apr 2026
Matthew Gibbons

Line chart showing 30 days of simulated daily visitors. Blue points show observed values scattered around the expected value of 1,000. An amber dashed horizontal line marks the point prediction at 1,000. A shaded amber band extends from roughly 938 to 1,062, representing the 95% prediction interval. Most but not all observations fall within the band.

How much should you trust a prediction?

foundations
thinking

A point prediction without uncertainty is like a deploy without monitoring. Prediction intervals tell you how much to trust a number, and whether the uncertainty is tight enough to act on.

23 Jan 2026
Matthew Gibbons

Line chart showing ten simulated time series of daily website visitors over 30 days. Ten blue lines each follow a different path, fluctuating between roughly 920 and 1,080 visitors per day. A dashed orange horizontal line marks the expected value of 1,000. Although every trace was generated from identical parameters, no two lines are the same, demonstrating that this variation is an intrinsic property of the Poisson model, not a modelling error.

Your model is wrong (and that’s the point)

foundations
thinking

Software engineers treat error as a bug, but in data science error is information. That shift changes everything.

3 Jan 2026
Matthew Gibbons
No matching items