P-ValuesP-Values

In the early 1920s, Ronald Fisher was the resident statistician at Rothamsted Experimental Station, an agricultural research center north of London. His day job was deciding whether one fertilizer or crop variety genuinely outperformed another, or whether the difference was just luck.

His answer appeared in the 1925 book 'Statistical Methods for Research Workers'. The p-value asks a sharp question: if there were truly no effect, how surprising would results at least this extreme be? A small p-value means the data would be a strange coincidence under the assumption of no effect.

Fisher suggested treating p below 0.05, a one-in-twenty coincidence, as convenient grounds to take a result seriously. He meant it as an informal working rule, not a law of nature. But the book became the standard manual for scientists who needed statistics without being statisticians, and 0.05 hardened into a bright line between publishable and forgettable.

The consequences still reverberate. Because careers hinge on crossing the threshold, researchers face pressure to slice data until something clears it, a practice now called p-hacking. The replication crisis that shook psychology and medicine in the 2010s traced much of its damage to ritualized threshold-chasing, and in 2016 the American Statistical Association took the unusual step of issuing a formal statement on what p-values do and do not mean.

A p-value is not the probability the hypothesis is true, and 0.05 carries no special physical significance. It is one piece of evidence, meant to be weighed alongside effect sizes, prior knowledge, and replication.

For machine learning practitioners the lesson is directly practical: whenever a model comparison or an A/B test declares a winner, the p-value machinery, with all its power and all its traps, is running underneath.