text

BERT — Bidirectional Encoder Representations from Transformers — is a transformer encoder pre-trained on two unsupervised objectives over raw text:

GPT — Generative Pre-Training — is a decoder-only transformer trained on an autoregressive (causal) language-modelling objective. Given a token sequence $x = (x_1, \dots, x_T)$, the network factorises the joint distribution left-to-right and maximises

GPT-2 retains GPT-1's decoder-only causal-LM objective

RoFormer replaces BERT's absolute (learned) position embedding with Rotary Position Embedding (RoPE) — a position-encoding scheme that injects relative-position information multiplicatively inside attention rather than additively at the input.

The Transformer replaces the recurrence of RNN-based seq2seq models with scaled dot-product attention, enabling fully parallel sequence processing. For query/key/value matrices $Q \in \mathbb{R}^{T_q \times d_k}$, $K \in \mathbb{R}^{T_k \times d_k}$, and $V \in \mathbb{R}^{T_k \times d_v}$, attention is