“Attention is all you need” – Paper Explained

Since the rise of ChatGPT, many similar AI models have emerged, including Gemini, Copilot, Claude, DeepSeek, and many others. These are known as LLMs (Large Language Models)—AI systems designed to understand and generate human language.

An LLM, or Large Language Model, can be oversimplified as a method to represent words, sentences, paragraphs, and the relationships between them in numbers, enabling computers to understand context, generate coherent responses, and perform a wide range of language-related tasks with remarkable accuracy.

And to achieve that remarkable accuracy, researchers conducted extensive studies and tried many methods until they discovered one that produced acceptable accuracy. By 2017, they found a method named “Attention Mechanism” that outperformed other existing methods, laying the foundation for all the popular AI we see today. This method is presented in a paper titled “Attention is All You Need,” which is published here: https://arxiv.org/pdf/1706.03762.

As you can see, the published paper was granted by Google. This means that researchers who work at Google found that method, not the CEO of AI companies! This is a misunderstanding that commonly happens when people cannot tell the difference between who discovered the method and who owns the company; sometimes it is the same person, and sometimes it is not!

1. What is Attention ?

Attention in “Attention is All You Need” is the “Attention Mechanism” – a method that enables an AI model to focus on important parts of a sentence instead of focusing on everything. Sounds human-like, right? But how does it feel for a computer?

Let’s say, given a computer the sentence:

“The cat sat on the mat because it was tired.”

Above sentence is understandable to human, but to a computer, what it sees is:

54 68 65 20 63 61 74 20 73 61 74 20 6f 6e 20 74 68 65 20 6d 61 74 20 62 65 63 61 75 73 65 20 69 74 20 77 61 73 20 74 69 72 65 64 2e 

In case you are curious, the numbers above are hexadecimals of each character forming the given sentence, and there is no such thing as “meaning” in a computer or an AI model.

As humans, when we read the word “it,” we immediately understand that it refers to “the cat.” This ability to connect one word to another comes naturally to us; the challenge is teaching a computer to make the same connection.

However, to a computer, every word is initially just a sequence of numbers. Those numbers alone do not tell the computer that “it” refers to “the cat.” The computer needs a way to represent these relationships mathematically—and that is exactly what the Attention Mechanism was designed to do.

We may easily refer to the word “Attention” as “focus” or “concentrate”, but to make it less confusing, the word “Attention” here is more about the word “attend” in this question: how much a word attend to other words in a sentence.

To be oversimplified, building an AI is very different from building software. In software, programmers write instructions to tell computers what to do, and the outcome is understood by humans. In AI, we do not write instructions because we cannot determine what those instructions should be; instead, we assume the outcome first and then let the computer find those instructions itself. That process is called “training” an AI.

And for “Attention Mechanism,” there is also a training phase that results in something called “attention score,” which can be oversimplified as the table below: (Note that these numbers are just examples, not actual scores in a real AI model)

These numbers tell the computer how much each word should influence the representation of the word “it”. In this example, “cat” receives the highest score. For each of the other words in the given sentence, there are also similar attention scores. Now, imagine scaling this to the scope of paragraphs and articles; the same mechanism is applied, and the computer can now see the relationships between every word, sentence, and paragraph by using attention scores.

The important point is that the computer is not actually reading or understanding language like a human. It is performing millions or billions of mathematical operations that calculate how strongly every word should be connected to every other word. These attention scores eventually help to build a Context-aware representation – a matrix that represents the meaning of given sentences or articles. From this Context-aware representation, the computer can generate text back in human language that eventually becomes the conversational abilities we experience in today’s AI, such as translation, question answering, summarizing, etc.

The experiment results in this paper again highlight a simple fact: the simple idea of allowing every word to “look at” every other word turned out to be so powerful that it became the foundation of modern LLMs.

2. Why Attention “is all you need” ?

Here, we will recall history in its oversimplified state to quickly understand the position of the Attention Mechanism in the whole LLM picture. In other words, we aim to understand the part “is all you need” in the paper title.

As we already know, the core idea of LLM is to find a Context-aware representation of a sentence, paragraph, or article in such a way that the outputted Context-aware representation has a method to represent most of the connections between words, similar to how the human brain perceives them. This process, technically, is called encoding. From the Context-aware representation, the computer can generate text that humans can understand; this process is called decoding. This pair of processes has established a common design pattern for most AI systems, named the Encoder-Decoder Architecture. The Encoder-Decoder Architecture can be simplified as below:

Input Sentence ->> Encoder ->> Context-aware representation ->> Decoder ->> Output Sentence

This model is also called the Sequence-to-Sequence model when it tries to convert a sequence (a sentence) into another sequence (another sentence).

2.1 How does the Encoder/Decoder encode/decode a sentence ?

As a simplest understanding, the Encoder and Decoder can be seen as two machines with a lot of parameters, ranging from millions to billions. Initially, these parameters are just random numbers. After billions of rounds of training, these parameters are gradually adjusted by a specific algorithm, which we call the Loss Function. Training an AI model means finding the best set of parameters that make the whole architecture produce the expected results. This best set of parameters, technically, is the trained AI model – the most important part of an AI system that determines how smart an AI is. These parameters are also called weights. When these weights are found, the Encoder and Decoder will use them to encode and decode. This is like after we have found a perfect set of parameters for a machine; we then keep using these values forever.

2.2 If Encoder & Decoder is machines, how does that machine look like inside?

For a while, both the Encoder part and the Decoder part in the Encoder-Decoder architecture were implemented by an RNN – Recurrent Neural Network. An RNN can look like this:

where each node (the circle) can look like this:

Each node performs a simple math operation given Inputs and Weights to produce an Output; there’s no magic there. The last Output (at the Output layer) is compared with the expected Output. Initially, the Output is always wrong, and this error is used as a metric to adjust the Weights. The computer does this loop billions of times until it produces the expected Output. This loop is called “training an AI.”

The final weights are the most optimal ones that allow the RNN to achieve the best results, making them the AI model’s most important assets. If the trained weights are leaked, other competitors can emerge without spending time and resources to produce those optimal weights.

2.3 Where is Attention in this process ?

For example, for the task of translating from English to German, the architecture will be:

English ->> RNN Encoder ->> Context-aware representation ->> RNN Decoder ->> German

The problem was that RNNs process words one by one and are known to have a few limitations:

• slow training (no parallelism)
• difficulty remembering long-range dependencies
• vanishing gradients

As researchers worked to improve the RNN Encoder–Decoder architecture, they discovered that the Decoder shouldn’t rely on a single context vector (the last output of the RNN Encoder), especially for long sentences. Instead, they allowed the Decoder to attend to all Encoder hidden states (intermediate calculations before reaching the last output) and compute a new context vector at each decoding step. After testing this method, they found that it significantly improved translation quality. However, the Encoder and Decoder were still RNNs. Thus, attention was considered an improvement to RNN Encoder–Decoder models, not a replacement.

Then, until this paper, researchers experimented with a new approach that only used the Attention Mechanism and got rid of RNNs. The architecture then became:

English ->> Attention Encoder ->> Context-aware representation ->> Attention Decoder ->> German

And then it worked so well in the translation task that researchers concluded: Attention alone is sufficient to model sequence relationships; RNNs and CNNs are no longer necessary. This is where the paper’s title comes from: “Attention Is All You Need“.

The researchers designed a new architecture that relied entirely on attention instead of RNNs. This architecture was named the “Transformer.”

2.4 How does a Transformer look like ?

For a simple explanation, the Transformer architecture can be seen as a machine with up to billions of tunable parameters. Those parameters are grouped into matrices of weights, each weight is a decimal number in the range between 0 and 1. These matrices include:

• WQ (Query Matrix Weights): used in Attention formula
• WK (Key Matrix Weights): used in Attention formula
• WV (Value Matrix Weights): used in Attention formula
• Word Embedding Matrix: used to convert human readable words to number representations
• Positional Embedding Matrix: represent each word’s position in a sentence.
• Feed-Forward Network Weights: this is where much of the model’s semantic knowledge, grammar patterns, language rules, and reasoning patterns are encoded.

Here is the diagram of a Transformer architecture. Above weight matrices can be seen as tunable parameters for blocks in this digram where:

• WQ, WK, WV are used to tune Multi-head Attention block (“Multi-head” term will be explained later)
• Word Embedding Matrix is used to tune Embedding block
• Positional Embedding Matrix is used to tune Positional encoding block
• Feed-Forward Network Weights is used to tune Position-wise FFM block
• Blocks Add & norm are mathematic transformations to keep the whole Transformer stable

3. How do they calculate Attention Scores ?

Before reading the steps below, re-enable your math brain; at the very least, recall how to compute the dot product from two matrices. (This is where school years count, haha): https://en.wikipedia.org/wiki/Dot_product

Attention score calculation occurs inside the Multi-head Attention block. The process of finding the attention scores for each word in a sentence is essentially the training phase of an AI model. As an oversimplified explanation, the steps to find attention scores are as follows:

3.1 Simplified Transformer Training Process

3.1.1 Prepare the training dataset

• Collect a large dataset of text (books, articles, blog posts, etc) and split it into tokens (aka words)
• For each input sentence, create Next Word Prediction tasks (see section 3.3 Next Word Prediction Task)

3.1.2 Initialize the model with random weights.

• Initialize random weights for WQ, WK, WV, Embedding, Positional Encoding, Feed Forward Network Weights.

3.1.3 Convert input tokens into embeddings.

• Each token (each word) is mapped to a dense vector (a set of numbers) called an embedding (call it E).

3.1.4 Generate Query (Q), Key (K), and Value (V) vectors

• For every token embedding E, and WQ, WK, WV, compute dot-product of matrices Q, K,V like so:
  Q = E . WQ
  K = E . WK
  V = E . WV
• Initially, these vectors are meaningless because the weight matrices are random.

3.1.5 Compute Attention

Attention Score for each token is computed by this formular:

Attention(Q, K, V)
    = softmax(QKᵀ / √d_k) V

This creates a context-aware representation for each token.

3.1.6 Predict the next token.

The decoder (or, in a decoder-only model like GPT, the final Transformer layers) uses context-aware representations to predict the probability of each word in the vocabulary being the next token.

3.1.7 Compute the loss.

• Compare the predicted probabilities with the correct next token from the training dataset.
• The difference is measured using a loss function (typically cross-entropy loss).

3.1.8 Update the weights.

• Backpropagation computes the gradients for every trainable parameter.
• An optimizer adjusts the weights—including WQ, WK, WV, embeddings, and all other parameters—to reduce loss.

3.1.9 Repeat for millions of training examples.

The model processes batch after batch, gradually learning meaningful embeddings, attention patterns, grammar, semantics, and world knowledge from the data.

3.1.10 Training ends when the model converges.

Training stops after a predefined number of epochs or when the loss no longer improves significantly. The learned weight matrices are then saved as the trained AI model.

3.2 How to understand the Attention(Q,K,V) formula ?

The paper names this formula Scaled Dot-Product Attention. This formula computes:

1. how strongly every word relates to one another,
2. normalizes those relationships into attention weights.
3. how to form the Context-aware representations from attention weights

This is essentially the core computation outlined in the Transformer paper:

3.2.1 Why Q, K, V ?

It is really abstract and hard to clearly explain what Q, K, and V represent. I don’t know what was in the researchers’ heads when they came up with this idea. That’s why we need researchers, not programmers, to find new ways! The most intuitive way to see it is: Each word is transformed into three different representations: one for asking (Query), one for being found (Key), and one for information (Value).

3.3 Next Word Prediction Task

One of the most important training tasks for a Large Language Model (LLM) is Next Word Prediction (more precisely, Next Token Prediction). During training, the model is given a large amount of text from books, websites, articles, source code, and other documents. It reads the text one token at a time and repeatedly plays a simple game: given all the previous tokens, predict what the next token will be.

For example, given the sentence:

The cat sat on the ____

The model does not know the missing token; instead, it predicts a probability for every token in its vocabulary. It may assign a high probability to “mat,” a lower probability to “floor,” and very low probabilities to unrelated words such as “computer.” The correct answer is then revealed, and the model measures how far its prediction was from the actual next token.

This prediction task is repeated billions or even trillions of times during training. After every prediction, the model computes a loss that measures the prediction error. Using an optimization algorithm such as gradient descent, the model slightly adjusts all of its trainable parameters—including the Word Embedding Matrix, Query/Key/Value matrices, and Feed-Forward Network weights—to make future predictions more accurate.

Although the objective sounds surprisingly simple, the model cannot succeed by merely memorizing the next word. To consistently predict the next token, it must gradually learn grammar, vocabulary, sentence structure, semantic relationships, facts about the world, programming syntax, reasoning patterns, and many other aspects of language. Through millions of optimization steps, the model becomes increasingly capable of understanding context and generating coherent text.

In other words, Next Token Prediction is not the final goal of the model—it is the training exercise through which the model learns language. Just as humans improve their skills by repeatedly solving practice problems, a Transformer improves by repeatedly predicting the next token and adjusting its internal weights based on its errors.

** There are 2 important concepts that remain: “Multi-head Attention” (slightly different from Attention) and Feed Forward Network, which are not explained in this post. There will be a next post for these two.**