Why Transformers Changed Everything
Before transformers, RNNs and LSTMs dominated sequence tasks. They processed data sequentially, which was slow and couldn't capture long-range dependencies well. Transformers process all tokens in parallel using self-attention.
Self-Attention Mechanism
Self-attention lets every token look at every other token to decide what's important.
python
class="text-purple-400 font-medium">import torch
class="text-purple-400 font-medium">import torch.nn.functional as F
class="text-purple-400 font-medium">def scaled_dot_product_attention(Q, K, V):
d_k = Q.size(-class="text-amber-300">1)
scores = torch.matmul(Q, K.transpose(-class="text-amber-300">2, -class="text-amber-300">1)) / (d_k ** class="text-amber-300">0.5)
weights = F.softmax(scores, dim=-class="text-amber-300">1)
return torch.matmul(weights, V)
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># Q, K, V come class="text-purple-400 font-medium">from linear projections of inputThe formula: Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) * V
- ●Q (Query): What am I looking for?
- ●K (Key): What do I contain?
- ●V (Value): What do I give?
Multi-Head Attention
Instead of one attention function, transformers use multiple heads that learn different relationships.
python
class MultiHeadAttention(nn.Module):
class="text-purple-400 font-medium">def __init__(self, d_model=class="text-amber-300">512, num_heads=class="text-amber-300">8):
super().__init__()
self.d_model = d_model
self.num_heads = num_heads
self.d_k = d_model // num_heads
self.W_q = nn.Linear(d_model, d_model)
self.W_k = nn.Linear(d_model, d_model)
self.W_v = nn.Linear(d_model, d_model)
self.W_o = nn.Linear(d_model, d_model)
class="text-purple-400 font-medium">def forward(self, x):
batch_size, seq_len, _ = x.shape
Q = self.W_q(x).view(batch_size, seq_len, self.num_heads, self.d_k).transpose(class="text-amber-300">1, class="text-amber-300">2)
K = self.W_k(x).view(batch_size, seq_len, self.num_heads, self.d_k).transpose(class="text-amber-300">1, class="text-amber-300">2)
V = self.W_v(x).view(batch_size, seq_len, self.num_heads, self.d_k).transpose(class="text-amber-300">1, class="text-amber-300">2)
attn = scaled_dot_product_attention(Q, K, V)
attn = attn.transpose(class="text-amber-300">1, class="text-amber-300">2).contiguous().view(batch_size, seq_len, self.d_model)
return self.W_o(attn)Positional Encoding
Transformers have no built-in sense of order. Positional encodings tell the model where each token is.
python
class="text-purple-400 font-medium">def positional_encoding(max_len, d_model):
pe = torch.zeros(max_len, d_model)
position = torch.arange(class="text-amber-300">0, max_len).unsqueeze(class="text-amber-300">1).float()
div_term = torch.exp(torch.arange(class="text-amber-300">0, d_model, class="text-amber-300">2).float() * -(torch.log(torch.tensor(class="text-amber-300">10000.0)) / d_model))
pe[:, class="text-amber-300">0::class="text-amber-300">2] = torch.sin(position * div_term)
pe[:, class="text-amber-300">1::class="text-amber-300">2] = torch.cos(position * div_term)
return peThe Transformer Block
Each transformer block has two main sub-layers:
- ●Multi-head self-attention
- ●Feed-forward network
Both use residual connections and layer normalization.
python
class TransformerBlock(nn.Module):
class="text-purple-400 font-medium">def __init__(self, d_model=class="text-amber-300">512, num_heads=class="text-amber-300">8, ff_dim=class="text-amber-300">2048):
super().__init__()
self.attention = MultiHeadAttention(d_model, num_heads)
self.norm1 = nn.LayerNorm(d_model)
self.norm2 = nn.LayerNorm(d_model)
self.ff = nn.Sequential(
nn.Linear(d_model, ff_dim),
nn.ReLU(),
nn.Linear(ff_dim, d_model)
)
class="text-purple-400 font-medium">def forward(self, x):
attn_out = self.attention(x)
x = self.norm1(x + attn_out)
ff_out = self.ff(x)
x = self.norm2(x + ff_out)
return xKey Takeaways
- ●Self-attention allows parallel processing of sequences
- ●Multi-head attention captures different types of relationships
- ●Positional encoding provides sequence order information
- ●The transformer architecture is the foundation of GPT, BERT, and all modern LLMs