BackpropagationBackpropagation

By the 1970s, neural networks were academically radioactive. Minsky and Papert's critique of the perceptron had convinced most of the field that the approach was a dead end, and researchers who kept working on multi-layer networks struggled to get published or funded at all.

The missing piece was a training method. A network with hidden layers could in principle represent almost anything, but nobody had a practical recipe for deciding how much each buried weight contributed to an error at the output. Versions of the answer had been found before, notably by Seppo Linnainmaa in 1970 and Paul Werbos in his 1974 thesis, but they sank without notice.

In 1986, David Rumelhart, Geoffrey Hinton, and Ronald Williams published 'Learning representations by back-propagating errors' in Nature. The algorithm used the chain rule of calculus to pass error signals backwards through the network, assigning blame layer by layer and nudging every weight in the right direction.

What made the paper land was not just the math but the demonstrations. Trained networks developed internal representations that nobody had designed, hidden units that came to encode meaningful features of the problem. Networks could finally learn XOR, the very function that had been used to bury the perceptron.

Adoption was still slow. Computers were underpowered, data was scarce, and training deep networks ran into the vanishing gradient problem, so through the 1990s and 2000s neural networks again fell out of fashion while other methods dominated.

The algorithm never changed; the hardware and datasets caught up. Backpropagation trained the networks that won ImageNet in 2012 and trains every large language model today. Hinton, along with Yoshua Bengio and Yann LeCun, received the 2018 Turing Award largely for refusing to give up on it.