Attention Is All You Need

"Attention Is All You Need" is a landmark 2017 research paper authored by eight scientists working at Google, responsible for expanding 2014 attention mechanisms proposed by Bahdanau et al. into a new deep learning architecture known as the transformer. The paper is considered by some to be a founding document for modern artificial intelligence, as transformers became the main architecture of large language models. At the time, the focus of the research was on improving Seq2seq techniques for machine translation, but even in their paper the authors saw the potential for other tasks like question answering and for what is now called multimodal Generative AI.

The paper's title is a reference to the song "All You Need Is Love" by the Beatles.

As of 2024, the paper has been cited more than 100,000 times.

Authors

The authors of the paper are: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. All eight authors were "Equal contributors" to the paper, and the name order was randomised. The Wired article highlights the group's diversity:

Six of the eight authors were born outside the United States; the other two are children of two green-card-carrying Germans who were temporarily in California and a first-generation American whose family had fled persecution, respectively.

By 2023, all eight authors had left Google and founded their own AI start-ups (except Łukasz Kaiser, who joined OpenAI).

References

1. All You Need Is Kill
2. All You Need Is Love
3. Ashish Vaswani
4. Attention (machine learning)
5. Transformer (deep learning architecture)
6. Vision transformer
7. YouTube Premium
8. All You Need Is...
9. Aidan Gomez
10. GPT-3
11. Inbox by Gmail
12. YouTube Shorts
13. T5 (language model)
14. All You Need Is Love: Meine Schwiegertochter ist ein Mann
15. Don't be evil
16. Google Brain
17. Google Messages
18. Google Translate
19. Frame rate
20. Motion interpolation