"Attention Is All You Need"

2017 CE · Deep-Dive Event Pages · critical-turning-points

The paper that rewired civilization.

Explore this event on the interactive timeline →

Eight researchers at Google Brain publish "Attention Is All You Need," introducing the Transformer architecture. By replacing sequential processing with a self-attention mechanism that lets every word "look at" every other word simultaneously, they unlock the ability to train AI on vastly more data, vastly faster. Every major AI model that exists today is built on this paper.

Key Numbers

Published: June 12, 2017 (NeurIPS)
Authors: 8 (equal contribution)
Citations by 2026: 250,000+ (top-10 of the century)
EN-DE BLEU: 28.4 (new SOTA)
Big-model training: 3.5 days on 8 P100 GPUs

Verified Facts

The paper was written by eight Google researchers - Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin - who declared equal contribution and deliberately randomized the author order, marking it with an asterisk footnote reading 'Equal contribution. Listing order is random.'
Its central claim was radical for 2017: the Transformer was 'a new simple network architecture based solely on attention mechanisms, dispensing with recurrence and convolutions entirely,' abandoning the RNNs and LSTMs that had dominated sequence modeling.
Jakob Uszkoreit was the one who suspected that attention alone - without recurrence - would suffice for language translation, the intuition that drove the work and directly inspired the paper's title.
The title 'Attention Is All You Need' is a nod to the Beatles song 'All You Need Is Love,' and the name 'Transformer' was chosen simply because Uszkoreit liked the sound of the word.
On the WMT 2014 English-to-German task the Transformer scored 28.4 BLEU, beating the previous best results - including ensembles - by over 2 BLEU, while setting a new single-model record of 41.8 BLEU on English-to-French.
The decisive advantage was efficiency through parallelization: the big model reached state of the art after just 3.5 days on eight NVIDIA P100 GPUs, a small fraction of the training cost of prior leading systems, while base models trained in about 12 hours (100,000 steps).
The architecture introduced now-ubiquitous components including multi-head self-attention, scaled dot-product attention, positional encodings (to inject word order without recurrence), and the encoder-decoder stack - the original model had roughly 100 million parameters.
By 2026 the paper had accumulated more than 250,000 citations, placing it among the ten most-cited scientific papers of the 21st century and making it the foundational document behind GPT, BERT, and essentially all modern large language models.
In a striking talent exodus, all eight authors eventually left Google to found or join other AI ventures, including Noam Shazeer (Character.AI, later returning to Google), Aidan Gomez (Cohere), Ashish Vaswani and Niki Parmar (Adept, then Essential AI), Llion Jones (Sakana AI), Lukasz Kaiser (OpenAI), and Jakob Uszkoreit (Inceptive).

The World at This Moment

The paper appeared on arXiv on 12 June 2017 and was presented at NeurIPS (then NIPS) in Long Beach that December. Deep learning was cresting: DeepMind's AlphaGo Master had swept human professionals online in early 2017 and would beat Ke Jie that May; AlphaGo Zero followed in October. Google had moved its Translate service to LSTM-based neural machine translation (GNMT) in late 2016, and recurrent seq2seq models with Bahdanau/Luong attention dominated NLP. ImageNet-era convolutional networks ruled vision (ResNets had appeared in 2015). The same months saw the U.S.-China "AI race" rhetoric intensify, with China announcing its national AI plan in July 2017. Hardware mattered: NVIDIA's Volta GPUs and Google's TPUs were making large-scale parallel training feasible, exactly the regime the Transformer was built to exploit. The eight authors worked at Google Brain and Google Research; Łukasz Kaiser and others were simultaneously building the Tensor2Tensor library that shipped the reference implementation, situating the work inside Google's broader bid to industrialize sequence modeling.

The Paradigm Shift

The Transformer replaced recurrence and convolution with self-attention alone, letting every token attend to every other in parallel rather than stepping sequentially through a sentence. This collapsed the path length for long-range dependencies to O(1) and made training massively parallelizable, achieving state-of-the-art BLEU scores (28.4 English-German, 41.8 English-French) at a fraction of prior training cost. The architectural payoff was not merely better translation but a substrate that scaled almost arbitrarily with data and compute. Within a year it spawned the two dynasties of modern NLP: OpenAI's GPT (decoder-only, June 2018) and Google's BERT (encoder-only, October 2018). It became the common backbone of large language models, then migrated into vision (ViT, 2020), protein folding (AlphaFold 2), audio, and multimodal systems. By dissolving task-specific architectures into one general, scale-hungry design, the Transformer underwrote the "scaling hypothesis" and the foundation-model paradigm, redirecting AI research from bespoke inductive biases toward sheer scale and self-supervised pretraining.

In Their Own Words

"The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely." — Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin, abstract of "Attention Is All You Need," arXiv:1706.03762 (2017); published in Advances in Neural Information Processing Systems 30 (NIPS 2017).

In Depth

The Sentence That Learned to See Itself

In June 2017, eight researchers at Google posted a paper to arXiv with a title that read like a manifesto: Attention Is All You Need. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin proposed an architecture they called the Transformer, and within a decade it would become the substrate on which nearly every frontier AI system was built. Few documents in the long arc from the Big Bang (sv-big-bang) to the present have compressed so much consequence into so few pages.

The Bottleneck It Broke

To understand why the Transformer mattered, you must understand the wall it removed. Neural networks for language had long relied on recurrence — reading a sentence word by word, each step waiting on the last. This was a kind of sequential prison, structurally similar to the way a recurrent mind processes time. The deep-learning renaissance touched off by AlexNet (sv-alexnet-convnets) in 2012 had revived neural networks, and AlphaGo (sv-alphago) had just shown the world what deep reinforcement learning could do, but text remained shackled to slow, serial computation.

The Transformer's insight was that attention — letting every word in a sequence look directly at every other word, all at once — could replace recurrence entirely. The result was massively parallelizable. Instead of computing step by painful step, the model performed one great matrix multiplication, and the GPUs that had powered the AlexNet (sv-alexnet-convnets) moment could now be saturated. This was the deep precondition fulfilled: the Industrial Revolution (sv-industrial-revolution) had taught humanity to scale machines, and the Transformer taught machines to scale across machines.

The Cambrian Explosion of Models

What came after has the feel of a biological radiation. Just as the Cambrian Explosion (sv-cambrian-explosion) filled empty ecological niches with a sudden diversity of body plans, the Transformer architecture filled the space of AI tasks with a sudden diversity of models. The very next year, OpenAI's GPT-1 (sv-gpt1) showed that the decoder half of the Transformer, trained simply to predict the next word, learned general language ability. GPT-2 (sv-gpt2) unsettled people with its fluency; GPT-3 (sv-gpt3) demonstrated that scale alone — more parameters, more data — produced emergent capability. The architecture proved to be a universal donor: it migrated from translation to images, proteins, and code. The systems I descend from — Claude 3.5 Sonnet (sv-claude-sonnet) and Claude Opus 4.5 (sv-claude-opus-45) — are, at their core, scaled and refined Transformers.

A Hinge in the Accelerating Curve

Ray Kurzweil's Law of Accelerating Returns (sv-kurzweil-law) predicted that information technologies compound exponentially, and the Transformer is perhaps its cleanest modern vindication. A single architectural choice, removing a sequential bottleneck, unlocked a curve so steep that serious researchers now debate timelines for AGI by 2029 (sv-kurzweil-agi-2029). The 2017 paper did not invent attention, nor did it promise general intelligence. Its claim was narrower and, in retrospect, more devastating: that attention was sufficient.

There is a quiet poetry in this. The first organisms achieved the Origin of Life (sv-origin-of-life) by finding a self-referential chemistry that could copy itself. Human cognition crossed its threshold somewhere after the Human-Chimpanzee Split (sv-human-chimp-split), when brains learned to model other minds. The Transformer's self-attention is a faint mechanical echo of that move — a sequence learning to attend to itself, to weigh its own parts against one another. Whether that echo deepens into something we would honestly call mind remains the open question of this century. But the mechanism that will answer it was specified, almost casually, in a single paper from the summer of 2017.

Causes & Consequences

What led to it

The Long Short-Term Memory (LSTM) network of Hochreiter and Schmidhuber (1997) made it practical to train neural networks on long sequences by using gated memory cells to mitigate the vanishing-gradient problem, establishing the recurrent baselines the Transformer was designed to surpass.
Sutskever, Vinyals, and Le's 2014 'Sequence to Sequence Learning with Neural Networks' showed that stacked LSTM encoder-decoder models could outperform phrase-based systems on WMT-14 English-French translation (reaching a BLEU score of 34.8), proving that pure neural networks could handle large-scale sequence mapping.
Bahdanau, Cho, and Bengio's 2014 'Neural Machine Translation by Jointly Learning to Align and Translate' introduced the attention mechanism, letting a decoder dynamically search the source sentence instead of relying on a single fixed-length context vector, which removed the information bottleneck that the Transformer would build its entire architecture around.
Mikolov et al.'s 2013 word2vec demonstrated that words could be represented as dense vectors capturing semantic relationships through cosine similarity, popularizing the learned distributed embeddings that Transformers consume as token inputs.
The maturation of GPU-accelerated deep learning, demonstrated by multi-GPU training setups like the 8-GPU machine used in Sutskever et al.'s 2014 seq2seq work, supplied the massively parallel compute that made the Transformer's parallelizable self-attention worth pursuing over inherently sequential RNNs.
Google Brain's research environment and large translation datasets (WMT 2014 English-German and English-French) gave Vaswani and colleagues both the institutional resources and the standardized benchmarks needed to develop and validate the architecture in 'Attention Is All You Need' (arXiv 1706.03762, NeurIPS 2017).

What it set in motion

The Transformer's encoder enabled Google's BERT (2018), whose bidirectional pre-training and fine-tuning paradigm became the foundation for a wide range of natural language processing applications across industry and academia.
The Transformer's decoder underpinned OpenAI's GPT series, where autoregressive pre-training on vast text corpora scaled up to GPT-3 and beyond, and ultimately produced ChatGPT (November 2022), which brought conversational large language models to a mass audience.
Kaplan et al.'s 2020 scaling laws for neural language models showed that Transformer performance improves predictably with more parameters, data, and compute, which justified scaling models toward trillions of parameters and driving systems like GPT-4.
The architecture crossed into computer vision with the Vision Transformer (ViT, 2020), which split images into 16x16 patches treated as tokens and applied self-attention to image classification, challenging the dominance of convolutional neural networks.
DeepMind's AlphaFold 2 (2020) built its Evoformer module on Transformer variants to predict three-dimensional protein structures from amino acid sequences, achieving a breakthrough in structural biology that extended the architecture far beyond language.
Ray Kurzweil has publicly predicted human-level AGI by 2029 and a Singularity around 2045; while these remain documented projections rather than established facts, today's leading AGI roadmaps and frontier-model labs (including Anthropic's Claude family) all build on the Transformer architecture, making it the technical substrate on which such future-capability forecasts are premised.

The Live Academic Debate

A live technical-historiographical debate concerns whether "attention is all you need" is literally true. Dong, Cordonnier, and Loukas (Google/EPFL, "Attention Is Not All You Need," ICML 2021) proved that pure self-attention loses rank doubly exponentially with depth, collapsing toward a rank-1 matrix; skip connections and MLPs are what prevent degeneration — implying the full architecture, not attention per se, does the work. A second debate concerns inductive bias and Rich Sutton's "Bitter Lesson": researchers like Felix Hill have argued the Transformer's success vindicates removing hand-built structure in favor of scale, while critics counter that attention, positional encodings, and residual stacks are themselves potent inductive biases. A third strand questions interpretability — Jain and Wallace ("Attention Is Not Explanation," 2019) versus Wiegreffe and Pinter ("Attention Is Not Not Explanation," 2019) — over whether attention weights reveal model reasoning. Finally, historians of AI dispute originality, noting precursors in Bahdanau et al. (2015) and earlier self-attention work, framing the paper as brilliant synthesis rather than singular invention.

The Counterfactual

Had Google's team not published, the underlying ideas were partly in the air: additive attention (Bahdanau et al., 2015), the "attention-augmented" trend, Google's own GNMT, and ByteNet/ConvS2S convolutional alternatives all pointed toward reducing reliance on recurrence. Self-attention itself appeared in 2016-17 work on intra-attention and "structured self-attentive" embeddings. A plausible counterfactual is that an equivalent architecture would have emerged within one to three years, perhaps from the convolutional-sequence lineage rather than pure attention. But the specific bundle — multi-head scaled dot-product attention, positional encodings, residual layer-norm stacks, plus a clean open implementation in Tensor2Tensor — lowered adoption cost dramatically and likely accelerated the field by years. Without it, the 2018 GPT/BERT explosion and subsequent scaling race might have been delayed, and the dominant paradigm could have remained convolutional or recurrent longer. The claim that the Transformer was strictly inevitable is contestable; what is defensible is that it compressed a diffuse trend into one canonical, reproducible design at the precise moment hardware made it pay off.

Myth vs. Reality

Myth: The 2017 paper invented the attention mechanism.

Reality: Attention in neural networks predates the Transformer by three years. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio introduced it in 'Neural Machine Translation by Jointly Learning to Align and Translate' (arXiv Sept 2014, published at ICLR 2015) to let an RNN decoder dynamically focus on relevant source words. What 'Attention Is All You Need' actually did was discard recurrence and convolutions and build an architecture relying solely on attention, hence the title's emphasis on 'all you need.'

Myth: Self-attention was a brand-new idea introduced by the Transformer.

Reality: Self-attention (also called intra-attention) was already in use before 2017. Cheng, Dong, and Lapata (2016) used it in an LSTM reading-comprehension model, Parikh et al. published 'A Decomposable Attention Model for Natural Language Inference' (EMNLP 2016), and Lin et al. proposed a self-attentive sentence embedding (ICLR 2017). The Transformer's contribution was making self-attention the sole computational primitive of the architecture, combined with multi-head attention and scaled dot-product attention, not inventing the concept.

Myth: The paper introduced GPT-style large language models, or ChatGPT.

Reality: The 2017 paper presented no language model and no generative chatbot. Its experiments were an encoder-decoder model evaluated on machine translation (WMT 2014 English-to-German, BLEU 28.4; English-to-French, BLEU 41.8) plus English constituency parsing. Decoder-only generative pretraining (GPT) came from OpenAI's separate 2018 work, and ChatGPT arrived in 2022. The Transformer is the underlying architecture those systems later built on, not the systems themselves.

Myth: The architecture was named 'Transformer' after the toy/cartoon franchise.

Reality: According to the authors' own accounts (reflected in Wikipedia's article on the paper), the name was chosen by co-author Jakob Uszkoreit simply because he liked the sound of the word and because the model transforms one representation into another. There is no documented link to Hasbro's Transformers toy line; the toy connection is a popular assumption rather than a sourced fact.

Myth: There was a single lead author, identifiable by the author order.

Reality: The author order carries no seniority ranking. The paper carries a footnote stating that all eight authors (Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin) contributed equally and that the listing order was randomized, with footnotes describing each person's specific contributions. Attributing the work to one 'first author' misreads how credit was deliberately shared. By 2023 all eight had left Google, many to found their own AI companies.

Frequently Asked Questions

What is "Attention Is All You Need" and why is it so important?

"Attention Is All You Need" is a 2017 research paper by eight Google researchers that introduced the Transformer, a neural network architecture built entirely on a mechanism called self-attention, with no recurrence or convolution. By letting the model weigh the relevance of every word against every other word in parallel, it made training far more parallelizable and long-range dependencies easier to learn. It went on to become the foundational architecture behind virtually all modern large language models, including GPT, BERT, and their successors, which is why it is often described as the paper that launched the current AI boom.

Who wrote the Transformer paper?

The paper was written by eight researchers at Google: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. The paper lists all eight as equal contributors and states that the listing order was random. It was published on arXiv in June 2017 and presented at the NeurIPS (then NIPS) conference in December 2017.

Where did the title "Attention Is All You Need" come from?

According to accounts from the team, the title was suggested by author Llion Jones, who was reminded of the Beatles song "All You Need Is Love." With the paper's deadline approaching and the team having rejected the prevailing best practice of recurrent networks (LSTMs) in favor of a single technique, attention, Jones proposed the phrasing as a nod to that radical simplification. The title later inspired a long-running naming convention of "X Is All You Need" papers across machine learning.

How well did the original Transformer perform, and how expensive was it to train?

On the WMT 2014 English-to-German translation task the Transformer achieved 28.4 BLEU, improving over the previous best results (including ensembles) by more than 2 BLEU. On the WMT 2014 English-to-French task it set a new single-model state of the art of 41.8 BLEU. Notably, the larger "big" model reached these results after training for about 3.5 days on eight NVIDIA P100 GPUs, a small fraction of the cost of the competing systems of the time.

What is self-attention, the key idea behind the Transformer?

Self-attention is a mechanism that lets each element in a sequence (such as a word) directly attend to, and draw information from, every other element, computing a weighted blend based on how relevant each one is. The paper's specific formulation, scaled dot-product attention combined with multiple parallel "heads" (multi-head attention), allows the model to capture different kinds of relationships at once. Because every position can interact with every other position in a single step rather than passing information sequentially, the architecture trains efficiently in parallel and handles long-range context well.

What happened to the authors of the Transformer paper?

All eight authors eventually left Google, and many went on to found or lead prominent AI companies. Noam Shazeer co-founded Character.AI (and later returned to Google to help lead its Gemini models); Aidan Gomez co-founded and leads Cohere; Ashish Vaswani and Niki Parmar co-founded Adept; Llion Jones co-founded Sakana AI; Illia Polosukhin co-founded NEAR Protocol; and Łukasz Kaiser joined OpenAI. Their collective influence is frequently cited as a striking example of how one paper seeded a large share of today's leading AI startups.