Transformer

The Transformer replaces the recurrence of RNN-based seq2seq models with scaled dot-product attention, enabling fully parallel sequence processing. For query/key/value matrices $Q \in \mathbb{R}^{T_q \times d_k}$, $K \in \mathbb{R}^{T_k \times d_k}$, and $V \in \mathbb{R}^{T_k \times d_v}$, attention is

where the $1/\sqrt{d_k}$ factor counteracts the variance of dot products in high dimensions and keeps softmax gradients well-conditioned.

Multi-head attention runs $h$ parallel attention heads on learned projections of $(Q, K, V)$ into $d_k = d_v = d_{\text{model}} / h$ subspaces, concatenates the outputs, and applies a final linear:

Each encoder layer stacks multi-head self-attention and a position-wise feed-forward network (with hidden width $d_{\text{ff}}=2048$), wrapped in residual connections and post-LayerNorm. Each decoder layer adds causal self-attention over previously generated targets and cross-attention over the encoder output.

Position information is injected via sinusoidal positional encodings of fixed (untrained) frequencies, which generalise to sequence lengths unseen during training. The paper's base model ($N=6$ encoder/decoder layers, $d_{\text{model}}=512$, $h=8$) achieves state-of-the-art WMT 2014 En-De / En-Fr BLEU while training in a fraction of the time of prior recurrent and convolutional seq2seq systems — the architectural foundation for essentially every modern language model.