Everything that’s swept the world since the 2022 release of ChatGPT — “the AI economy”; data centers; chatbots acting as best friends, doctors, and mathematicians; talk of existential risk — it all starts, in some sense, with an intern in a University of Montreal lab named Dzmitry Bahdanau.

It was 2014. The 23-year-old Belarusian, who had been coding competitively since he was a kid but hadn’t done academic research before, was asked to help figure out how to get a computer to translate sentences from English to French. Bahdanau was working with artificial neural networks, which had already shown promise in pattern-recognition tasks such as deciphering handwriting and identifying objects in photos. The networks learn to recognize patterns by strengthening and weakening connections between nodes in response to training data. Bahdanau quietly doubted their potential to achieve anything like human-level intelligence, but his adviser, the computer scientist Yoshua Bengio, and the postdoc on the project, Kyunghyun Cho, were believers. And translation was widely seen as a formidable and interesting problem for neural networks to try to solve.

Translation is not as simple as swapping each English word for a French one. Languages have different grammatical structures and are riddled with idioms and connotations, all of which had long stymied attempts to automate the process. You can’t simply implement rules of translation because there are none. But neural networks seemed promising because they learn by example. They can be trained on huge databases of paired English and French sentences to, eventually, absorb the quasi-rules, exceptions, and subtleties.

Results had been mediocre, though; the longer the sentence, the worse neural networks performed. That’s because researchers were trying to cram each sentence’s data into a fixed amount of memory before decoding the sentence in the target language. It was a bottleneck that limited how complex and sophisticated translations could be. The network architecture just wasn’t right.

So Bahdanau tried something different. “It was just obviously the right thing to do — one of these things that happens once in a while in science where it’s just very clear to someone who is not very experienced in the field,” he recalled, speaking to me from Montreal, where he still works in AI. “And it just worked. From the very first attempt.”

Bahdanau’s trick, a mathematical operation that he, Cho, and Bengio referred to as “attention,” enabled their neural network to translate long sentences just as well as short ones. That meant the approach was scalable. Other AI researchers at the time quickly integrated attention into their neural network designs. Then, in 2017, a team at Google devised a new kind of neural network architecture called a transformer that supercharged the attention method and stripped away everything else. The team’s paper, “Attention Is All You Need,” reported that transformers achieved state-of-the-art translation while requiring a fraction of the computation needed to train prior models. The paper, which has since been cited more than 250,000 times, was a watershed moment in AI. Large language models such as ChatGPT and Claude followed.

Given the staggering power that’s been unleashed, it’s worth asking what this attention mechanism is exactly, why it is “all you need,” and why machine translation was the task that unlocked it all. Attention is a way for us to understand how a single AI model can converse with us, diagnose diseases, generate realistic videos, solve long-standing math problems, and more. Is it somehow at the heart of intelligence — artificial and otherwise?