RoFormer

RoFormer replaces BERT's absolute (learned) position embedding with Rotary Position Embedding (RoPE) — a position-encoding scheme that injects relative-position information multiplicatively inside attention rather than additively at the input.

Consider a query/key vector $x \in \mathbb{R}^{d}$ at position $m$. RoPE groups the $d$ features into $d/2$ complex coordinates and rotates the $i$-th pair $(x_{2i}, x_{2i+1})$ by angle $m \theta_i$, where

and base (default $10\,000$) controls the spectrum of rotational frequencies. In complex form $z_i = x_{2i} + j\, x_{2i+1}$, the rotation is just $z_i \mapsto z_i\, e^{j m \theta_i}$.

The crucial property is that the attention dot product becomes a function of the relative offset $m - n$ only:

so RoPE encodes relative positions without any extra parameters and without an additive bias term. This grants three practical benefits: (i) it generalises to longer sequences than seen during training (with mild quality decay), (ii) it composes naturally with linear attention variants because it acts on $Q, K$ before the softmax, and (iii) it requires the per-head dimension $d_{\text{head}} = d_{\text{model}} / h$ to be even — enforced by __post_init__ above. RoFormer otherwise mirrors BERT-base ($L=12$, $H=768$, $A=12$) and is the de-facto position-encoding choice for modern LLaMA-/Mistral-style LLMs.