The TransformerThe Transformer

In 2017, a team of Google researchers published 'Attention Is All You Need,' introducing the Transformer. It has since become the backbone of nearly every large language model, image generator and even protein-folding system.

Before the Transformer, the best language models processed text one word at a time using recurrent networks, which were slow and struggled to connect distant words. The Transformer threw out recurrence entirely and relied on a mechanism called attention to look at all the words at once.

Attention lets the model weigh how much each word should focus on every other word. To interpret the word 'it' in a sentence, the model can attend directly to the noun it refers to, no matter how far away, learning relationships across an entire passage in parallel.

Because everything happened in parallel rather than in sequence, Transformers trained far faster on modern hardware, and they kept getting better as researchers made them larger. That scalability turned out to be the whole game.

Ironically, the paper was written for machine translation, a fairly narrow task. Its authors could hardly have known they had drawn the blueprint for GPT, BERT, AlphaFold and the generative AI boom that followed.