Attention Is All You Need: Understanding the Transformer Architecture

Breye Foka

Cover Image for Attention Is All You Need: Understanding the Transformer Architecture

Breye Foka

April 10, 2026

🏁 Introduction: The Architecture That Changed Everything

In 2017, researchers at Google published a paper titled "Attention Is All You Need." The title was bold. The claim was bolder. Before this paper, sequence modeling relied on Recurrent Neural Networks (RNNs) — models that processed tokens one by one, maintaining a hidden state. This sequential nature made them slow to train and hard to parallelize.

The Transformer abolished the recurrence. Instead, it used a mechanism called self-attention to process all tokens simultaneously, letting every position directly attend to every other position. The result was faster training, better long-range dependency modeling, and a model that scaled incredibly well with data and compute.

Today, Transformers power ChatGPT, Gemini, Claude, GitHub Copilot, AlphaFold 2 (protein structure prediction), and even modern computer vision models. Understanding how they work isn't optional for anyone serious about ML.

⚠️ The Problem with RNNs

A vanilla RNN processes a sequence $x_1, x_2, \ldots, x_T$ step by step:

$h_t = \tanh(W_h h_{t-1} + W_x x_t + b)$

The problems:

Sequential bottleneck: $h_t$ depends on $h_{t-1}$ , which depends on $h_{t-2}$ , ... Can't parallelize.
Vanishing gradients: Information from early tokens gets diluted over many timesteps.
Fixed-size hidden state: The entire context of a long sequence must be compressed into one vector.

LSTMs and GRUs helped with vanishing gradients, but the sequential bottleneck remained. Something fundamentally different was needed.

🔍 The Core Idea: Self-Attention

Self-attention lets each token in a sequence directly "look at" every other token and decide how much to attend to it.

For a sequence of $n$ tokens, each represented as a vector, self-attention outputs a new representation for each token that is a weighted sum of all token values, where the weights reflect relevance.

Queries, Keys, and Values

The mechanism uses three learned linear projections of each input embedding:

$Q = XW^Q, \quad K = XW^K, \quad V = XW^V$

Where $X \in \mathbb{R}^{n \times d_{model}}$ is the input sequence and $W^Q, W^K, W^V \in \mathbb{R}^{d_{model} \times d_k}$ are learned weight matrices.

Think of it as an information retrieval system:

Query (Q): "What am I looking for?"
Key (K): "What do I have to offer?"
Value (V): "Here's the actual content I'll contribute"

The Attention Score

The attention weight between position $i$ and position $j$ is computed as:

$\text{score}(i, j) = \frac{q_i \cdot k_j}{\sqrt{d_k}}$

The $\sqrt{d_k}$ scaling prevents dot products from growing large (which would push softmax into regions of very small gradients).

Scaled Dot-Product Attention

Putting it together for the full sequence:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) V$

In matrix form:

$QK^T \in \mathbb{R}^{n \times n}$ — the full attention score matrix
After softmax: each row sums to 1 (a probability distribution over positions)
Multiplying by $V$ : each output is a weighted average of all value vectors

import numpy as np

def scaled_dot_product_attention(Q, K, V, mask=None):
    d_k = Q.shape[-1]
    
    # Compute attention scores
    scores = Q @ K.transpose(-2, -1) / np.sqrt(d_k)  # (n, n)
    
    # Optional mask (for causal/autoregressive attention)
    if mask is not None:
        scores = scores + mask * -1e9
    
    # Softmax to get attention weights
    weights = np.exp(scores - scores.max(axis=-1, keepdims=True))
    weights = weights / weights.sum(axis=-1, keepdims=True)
    
    # Weighted sum of values
    output = weights @ V  # (n, d_v)
    return output, weights

🌐 Multi-Head Attention

A single attention head captures one type of relationship between tokens. Multi-head attention runs $h$ attention heads in parallel, each learning different relationship patterns:

$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h) W^O$

$\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V)$

Each head has its own $W^Q_i, W^K_i, W^V_i$ of dimension $d_{model}/h$ , keeping the total computation similar to single-head attention.

Why multiple heads? Different heads learn to attend to different aspects simultaneously:

One head might track syntactic agreement (subject-verb)
Another might track coreference ("the animal... it...")
Another might capture positional relationships

class MultiHeadAttention:
    def __init__(self, d_model, num_heads):
        self.h = num_heads
        self.d_k = d_model // num_heads
        d = d_model
        # Weight matrices (simplified — normally these are nn.Linear layers)
        self.W_Q = np.random.randn(d, d) * 0.01
        self.W_K = np.random.randn(d, d) * 0.01
        self.W_V = np.random.randn(d, d) * 0.01
        self.W_O = np.random.randn(d, d) * 0.01

    def forward(self, X):
        n, d = X.shape
        Q = X @ self.W_Q
        K = X @ self.W_K
        V = X @ self.W_V

        # Reshape for multi-head: (n, h, d_k)
        Q = Q.reshape(n, self.h, self.d_k)
        K = K.reshape(n, self.h, self.d_k)
        V = V.reshape(n, self.h, self.d_k)

        # Attend per head, then concatenate
        heads = []
        for i in range(self.h):
            head, _ = scaled_dot_product_attention(Q[:, i], K[:, i], V[:, i])
            heads.append(head)

        concat = np.concatenate(heads, axis=-1)  # (n, d_model)
        return concat @ self.W_O

🧭 Positional Encoding

Self-attention is permutation invariant — shuffle the tokens and the attention scores change, but there's no inherent notion of order. We need to inject positional information.

The original paper uses sinusoidal positional encodings:

$PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{model}}}\right)$

$PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)$

These are added (not concatenated) to the input embeddings. The choice of sinusoids is deliberate: they allow the model to attend to relative positions, and the pattern generalizes to sequence lengths not seen during training.

Modern variants learn positional encodings or use Rotary Position Embedding (RoPE) — but the intuition remains: we need to tell the model where each token is.

🏛️ The Full Transformer Block

A single Transformer encoder block stacks these components:

Input
  │
  ├─► Multi-Head Self-Attention → Add & Norm (residual)
  │
  └─► Feed-Forward Network (2-layer MLP) → Add & Norm (residual)
  │
Output

The Feed-Forward Network (FFN) is a simple 2-layer MLP applied independently to each position:

$\text{FFN}(x) = \max(0, xW_1 + b_1)W_2 + b_2$

Where the inner dimension is typically $4 \times d_{model}$ . This is where most of the "knowledge" is thought to be stored in large language models.

Residual connections (the "Add" steps) are critical — they let gradients flow directly backwards, enabling training of very deep Transformers (the same idea as ResNets).

Layer Normalization stabilizes training by normalizing activations across the feature dimension:

$\text{LayerNorm}(x) = \frac{x - \mu}{\sigma + \epsilon} \cdot \gamma + \beta$

📐 Encoder vs Decoder

The original Transformer (for machine translation) has both:

Encoder: Processes the input sequence. All tokens attend to all other tokens (bidirectional attention). Used in models like BERT.

Decoder: Generates the output token by token. Uses causal (masked) attention — token $i$ can only attend to positions $\leq i$ (you can't look at future tokens when generating). Used in GPT-style models.

The causal mask is an upper-triangular matrix of $-\infty$ values added to attention scores before softmax, zeroing out future attention weights.

📊 Attention Complexity

Self-attention has $O(n^2 \cdot d)$ complexity — quadratic in sequence length. For $n = 1000$ tokens, that's a million attention score computations. For $n = 100,000$ (long documents, DNA sequences), this becomes prohibitive.

This has spawned an entire field of efficient attention variants:

Sparse Attention (Longformer, BigBird): attend only to nearby tokens + a few global ones
Linear Attention (Performer): approximate full attention in $O(n)$
Flash Attention: same math, much faster via I/O-aware GPU computation

🌍 Why Transformers Dominate

The Transformer's success stems from three properties:

Parallelism: Unlike RNNs, the entire sequence is processed at once — modern GPU hardware loves this
Direct long-range attention: Any two tokens can interact in a single layer, without information passing through intermediate steps
Scalability: Performance improves predictably with more data, more parameters, and more compute ("scaling laws")

That last property led to GPT-3 (175B parameters), GPT-4, and the current era of large language models.

🧠 Final Thoughts

We've now traced the arc from a single perceptron to the architecture behind today's most powerful AI systems:

| Building Block | Where We Covered It | |---|---| | Perceptron & Activation Functions | MIT 6.S191 Deep Learning post | | Backpropagation & Gradient Descent | Neural Network from Scratch post | | Loss Functions & Optimization | Linear & Logistic Regression posts | | Spatial Feature Learning | CNN post | | Sequence Modeling & Attention | This post |

The field moves fast, but the foundations are stable. Every new architecture — diffusion models, state space models, mixture of experts — is built from the same mathematical primitives we've been learning.

Stay curious, keep building, and don't let the acronyms intimidate you.

👨‍💻 Resources

Original paper: Attention Is All You Need (Vaswani et al., 2017)
Illustrated guide: The Illustrated Transformer by Jay Alammar
Hands-on: Andrej Karpathy's nanoGPT

Written by Breye.