GPT-2

GPT-2 retains GPT-1's decoder-only causal-LM objective

but introduces three architectural refinements that have since become standard for autoregressive transformers. Pre-LayerNorm moves the LayerNorm to the input of every sub-block (attention and MLP), so each block computes $h \leftarrow h + f(\mathrm{LN}(h))$ instead of $h \leftarrow \mathrm{LN}(h + f(h))$; combined with a final LayerNorm (ln_f) after the last block, this yields much more stable optimisation at depth. Residual projection weights are initialised with an extra $1/\sqrt{2N}$ factor (where $N$ is the layer count), which keeps activation variance roughly invariant to depth.

The tokenizer switches to a byte-level BPE with a 50257-symbol vocabulary (50256 merges plus the <|endoftext|> sentinel), and context length doubles from 512 to 1024. The released family spans four sizes — small (124M, the defaults above), medium (355M), large (774M), and xl (1.5B) — by scaling $(\text{hidden\_size}, \text{num\_hidden\_layers}, \text{num\_attention\_heads})$ while holding intermediate_size = 4 \cdot \text{hidden\_size} and head-dim = 64 fixed.

The paper's headline finding is that a sufficiently large LM trained on WebText (~40 GB scraped from outbound Reddit links) performs competitive zero-shot transfer across reading comprehension, translation, summarisation, and QA — without any task-specific fine-tuning. This is the empirical seed of in-context learning and the modern prompting paradigm.