“Attention is all you need” – Paper Explained

Since the rise of ChatGPT, many similar AI models have emerged, including Gemini, Copilot, Claude, DeepSeek, and many others. These are known as LLMs (Large Language Models) – AI systems designed to understand and generate human language.

An LLM, or Large Language Model, as an oversimplified state, can be seen as a method to represent words, sentences, paragraphs, and the relationships between them in numbers, enabling computer to understand context, generate coherent responses, and perform a wide range of language-related tasks with remarkable accuracy.

And to achieve that remarkable accuracy, researchers already made a lot of researches, try many methods until find out a method that produce acceptable accuracy. Then until 2017, they found a method named “Attention Mechanism” that outperform other existing methods, and then, put a first stone for all popular AI we see today. This method is presented in a paper with title “Attention is all you need” and is published here: https://arxiv.org/pdf/1706.03762 .

As you can see, the published paper was granted by Google. It means that researchers who work at Google found that method, not the CEO of AI companies !!! This is a misunderstand commonly happens when people can not tell the difference between who found the way and who own a company, sometime it is the same person, sometime it does not!

1. What is Attention ?

“Attention” in “Attention is all you need” is “Attention Mechanism” – a method that enable an AI model to focus on importance parts of a sentence instead of focusing on everything. Sounds like human ? But how it feel like for a computer ?

Let’s say, given a computer the sentence:

“The cat sat on the mat because it was tired.”

Above sentence is understandable to human, but to a computer, what it sees is:

54 68 65 20 63 61 74 20 73 61 74 20 6f 6e 20 74 68 65 20 6d 61 74 20 62 65 63 61 75 73 65 20 69 74 20 77 61 73 20 74 69 72 65 64 2e 

In case you are curios, above numbers are hexadecimals of each character forming the given sentence, and there is no such a thing called “meaning” in a computer, or in an AI model.

As humans, when we read the word “it,” we immediately understand that it refers to “the cat.” This ability to connect one word to another comes naturally to us. The challenge is teaching a computer to make the same connection.

However, to a computer, every word is initially just a sequence of numbers. Those numbers alone do not tell the computer that “it” refers to “the cat.” The computer needs a way to represent these relationships mathematically—and that is exactly what the Attention Mechanism was designed to do.

We may easily refer word “Attention” to “focus” or “concentrate”, but to make it less confusing, word “Attention” here is more about word “attend” in this question: how much a word attend to another words in a sentence .

To be oversimplified, building an AI is a lot different from building a software. In software, programmers write instructions to tell computers what to do, and the outcome is made sense by human. In AI, we do not write instructions because we can’t tell which instructions should be, but we assume the outcome first, then let the computer find that instructions itself. That process is called “training” an AI.

And for “Attention Mechanism”, there is also a training phase that as a result, it can produce something called “attention score” which can be again oversimplified as below table: (note that these number is just examples, not real score in real AI model)

These numbers tell the computer how much each word should influence the representation of word“it”. In this example, “cat” receives the highest score. And for each other words in the given sentence, there are also similar attention scores like this as well. Now imagine, scale this to paragraph and article scope, the same mechanism is applied and computer now can see the relationship between every words, sentences and paragraphs by using attention scores.

The important point is that the computer is not actually reading or understanding language like a human. It is performing millions or billions of mathematical operations that calculate how strongly every word should be connected to every other word. These attention scores eventually help to build a Context-aware representation – a matrix that represent for meaning of given sentences or articles. From this Context-aware representation, computer can generate back texts in human language that is eventually become conversational abilities that we experience on today AI such as: translation, question answering, summarizing, etc

The experiment result in this paper again highlight a simple fact that: the simple idea – allowing every word to “look at” every other words – turned out to be so powerful that it became the foundation of modern LLMs.

2. Why Attention “is all you need” ?

Here, we will recall history in its oversimplified state to quickly understand the position of Attention Mechanism in the whole LLM picture. In another word, we try to understand the part “is all you need” in the paper title.

As we already know, the core idea of LLM is to find a Context-aware representation of a sentence, paragraph or articles in a such way that the outputted Context-aware representation has a way to represent most of connections between words, as much as how human brain can see. This process, technically, is called Encode. And from Context-aware representation, computer can generate text that human can understand, this process is call Decode. This pair of process found a common design pattern for most of AI system, named: Encoder-Decoder Architecture. The Encoder-Decoder Architecture can be simplify as below:

Input Sentence ->> Encoder ->> Context-aware representation ->> Decoder ->> Output Sentence

This model is also called Sequence-to-Sequence model when it try to convert a sequence (a sentence) to another sequence (another sentence).

2.1 How does the Encoder/Decoder encode/decode a sentence ?

As a simplest understanding, Encoder & Decoder can be seen as 2 machines with a lot of parameters, up to millions to billions parameters. Initially, these parameters are just random numbers. After billions round of training, these parameters are gradually adjusted by a specific algorithm – which we call it the Loss Function. Training an AI model means to find the best set of parameters that make the whole architecture produce the expected results. This the best set of parameters, technically, is the trained AI model – the most important part of an AI system that determine how smart an AI is. These parameters also are called as weights. When these weights are found, the Encoder & Decoder will use it to encode & decode. This is like after we found a perfect set of parameters on a machine, we then keep using these values forever.

2.2 If Encoder & Decoder is machines, how does that machine look like inside?

For a while, both Encoder part & Decoder part in Encoder-Decoder architecture is implemented by a RNN – Recurrent Neural Network. A RNN can look like this:

where each node (the circle) can look like this:

Each node does a simple math given Inputs & Weights and produce an Output, no magic there. The last Output (at Output layer) is compared with the expected Output. Initially, Output is always wrong and the wrong here is used as a metric to adjust Weights. And computer do that loop billions of time until it produce the expected Output. This loop is called “training an AI”.

The final Weights is the most optimal one that makes RNN achieve best results will be the AI model – the most important assets of an AI company. If trained Weights is leaked, other competitors can emerge without spending time & resources to produce that optimal Weights.

2.3 Where is Attention in this process ?

For example, for task translation from English to German, the architecture will be:

English ->> RNN Encoder ->> Context-aware representation ->> RNN Decoder ->> German

The problem was that, RNNs process words one by one and are known to have a few limitations:

• slow training (no parallelism)
• difficulty remembering long-range dependencies
• vanishing gradients

As researchers worked to improve the RNN Encoder–Decoder architecture, they discovered that the Decoder shouldn’t rely on a single context vector (the last output of RNN Encoder), especially for long sentences. Instead, they allowed the Decoder to attend to all Encoder hidden states (intermediate calculations before reaching to the last output) and compute a new context vector at each decoding step. Then after tested this method, they found that this significantly improved translation quality. However, the Encoder and Decoder were still RNNs. So attention was considered an improvement to RNN Encoder–Decoder models, not a replacement.

Then, until this paper, researchers experimented a new approach that only use the Attention Mechanism, and got rid of RNNs. The architecture then became:

English ->> Attention Encoder ->> Context-aware representation ->> Attention Decoder ->> German

And then it worked so well in translation task and so that, researchers concluded: Attention alone is sufficient to model sequence relationships; RNNs and CNNs are no longer necessary. This is where the paper’s title comes from: “Attention Is All You Need“.

The researchers then designed a new architecture that relied entirely on Attention instead of RNN. This architecture was named the “Transformer“.

2.4 How does a Transformer look like ?

For a simple explanation, the Transformer architecture can be seen as a machine with up to billions tunable parameters. Those parameters are grouped into matrices of weights, each weight is a decimal number in range between 0 an 1. These matrices includes:

• WQ (Query Matrix Weights): used in Attention formula
• WK (Key Matrix Weights): used in Attention formula
• WV (Value Matrix Weights): used in Attention formula
• Word Embedding Matrix: used to convert human readable words to number representations
• Positional Embedding Matrix: represent each word’s position in a sentence.
• Feed-Forward Network Weights: this is where much of the model’s semantic knowledge, grammar patterns, language rules, and reasoning patterns are encoded.

Here is the diagram of a Transformer architecture. Above weight matrices can be seen as tunable parameters for blocks in this digram where:

• WQ, WK, WV are used to tune Multi-head Attention block (“Multi-head” term will be explained later)
• Word Embedding Matrix is used to tune Embedding block
• Positional Embedding Matrix is used to tune Positional encoding block
• Feed-Forward Network Weights is used to tune Position-wise FFM block
• Blocks Add & norm are mathematic transformations to keep the whole Transformer stable

3. How do they calculate Attention Scores ?

Before reading below steps, load your math brain back, at least, remember how to compute Dot-Product from 2 matrices. (this is where school years count, haha ): https://en.wikipedia.org/wiki/Dot_product

Attention Score Calculation happens inside the Multi-head Attention block. The process to find the attention scores for each word in a sentence is actually the training phase of an AI model. As an oversimplified explanation, steps to find attention scores is like so:

3.1 Simplified Transformer Training Process

3.1.1 Prepare the training dataset

• Collect a large dataset of text (books, articles, blog posts, etc) and split it into tokens (aka words)
• For each input sentence, create Next Word Prediction tasks (see section 3.3 Next Word Prediction Task)

3.1.2 Initialize the model with random weights.

• Initialize random weights for WQ, WK, WV, Embedding, Positional Encoding, Feed Forward Network Weights.

3.1.3 Convert input tokens into embeddings.

• Each token (each word) is mapped to a dense vector (a set of numbers) called an embedding (call it E).

3.1.4 Generate Query (Q), Key (K), and Value (V) vectors

• For every token embedding E, and WQ, WK, WV, compute dot-product of matrices Q, K,V like so:
  Q = E . WQ
  K = E . WK
  V = E . WV
• Initially, these vectors are meaningless because the weight matrices are random.

3.1.5 Compute Attention

Attention Score for each token is computed by this formular:

Attention(Q, K, V)
    = softmax(QKᵀ / √d_k) V

This produces a context-aware representation for every token.

3.1.6 Predict the next token.

The decoder (or, in a decoder-only model like GPT, the final Transformer layers) uses the context-aware representations to predict the probability of every word in the vocabulary being the next token.

3.1.7 Compute the loss.

• Compare the predicted probabilities with the correct next token from the training dataset.
• The difference is measured using a loss function (typically cross-entropy loss).

3.1.8 Update the weights.

• Backpropagation computes gradients for every trainable parameter.
• An optimizer adjusts the weights—including WQ, WK, WV, embeddings, and all other parameters—to reduce the loss.

3.1.9 Repeat for millions of training examples.

The model processes batch after batch, gradually learning meaningful embeddings, attention patterns, grammar, semantics, and world knowledge from the data.

3.1.10 Training ends when the model converges.

Training stops after a predefined number of epochs or when the loss no longer improves significantly. The learned weight matrices are then saved as the trained AI model.

3.2 How to understand the Attention(Q,K,V) formula ?

The paper named this formula Scaled Dot-Product Attention. This formula is to computes:

1. how strongly every word relates to every other word,
2. normalizes those relationships into attention weights,
3. how to forming the Context-aware representations from attention weights

This is essentially the core computation inside the Transformer paper:

3.2.1 Why Q, K, V ?

It is really abstract and hard to clearly explain what Q, K, V represents for. I don’t know what in researchers heads that came up to this idea. It is why we need researchers, not programmers to found the new way! The most intuitive way to see it is: Each word is transformed into three different representations: one for asking (Query), one for being found (Key), and one for information (Value).

3.3 Next Word Prediction Task

One of the most important training tasks for a Large Language Model (LLM) is Next Word Prediction (more precisely, Next Token Prediction). During training, the model is given a large amount of text from books, websites, articles, source code, and other documents. It reads the text one token at a time and repeatedly plays a simple game: given all the previous tokens, predict what the next token should be.

For example, consider the sentence:

The cat sat on the ____

The model does not know the missing token. Instead, it predicts a probability for every token in its vocabulary. It may assign a high probability to “mat”, a lower probability to “floor”, and very low probabilities to unrelated words such as “computer”. The correct answer is then revealed, and the model measures how far its prediction was from the actual next token.

This prediction task is repeated billions or even trillions of times during training. After every prediction, the model computes a loss, which measures the prediction error. Using an optimization algorithm such as gradient descent, the model slightly adjusts all of its trainable parameters—including the Word Embedding Matrix, Query/Key/Value matrices, and Feed-Forward Network weights—to make future predictions more accurate.

Although the objective sounds surprisingly simple, the model cannot succeed by merely memorizing the next word. To consistently predict the next token, it must gradually learn grammar, vocabulary, sentence structure, semantic relationships, facts about the world, programming syntax, reasoning patterns, and many other aspects of language. Over millions of optimization steps, the model becomes increasingly capable of understanding context and generating coherent text.

In other words, Next Token Prediction is not the final goal of the model—it is the training exercise through which the model learns language. Just as humans improve their skills by repeatedly solving practice problems, a Transformer improves by repeatedly predicting the next token and adjusting its internal weights based on its mistakes.

** There are remain 2 important concepts that are “Multi-head Attention” (slightly different from Attention) and Feed Forward Network which is not explained in this post. There will be next post for these two.