BERT

BERT — Bidirectional Encoder Representations from Transformers — is a transformer encoder pre-trained on two unsupervised objectives over raw text:

1. Masked Language Modelling (MLM). A random 15% of input tokens are replaced with a [MASK] placeholder; the model must predict the original token from its bidirectional context. Formally, for masked positions $\mathcal{M}$,

Unlike the left-to-right LMs that preceded it, every token's prediction conditions on both left and right context — hence bidirectional.

1. Next-Sentence Prediction (NSP). Two segments $A$ and $B$ are concatenated with separator tokens; the model reads the pooled [CLS] representation and predicts whether $B$ was the actual next sentence in the corpus or a random other sentence. This objective gives the model a coarse notion of inter-sentence coherence used by downstream sentence-pair tasks (QA, NLI).

After pre-training on BookCorpus + English Wikipedia, the same encoder is fine-tuned on individual downstream tasks by adding a thin task-specific head on top of either the [CLS] token representation (classification) or all per-token representations (tagging, span prediction). This "pre-train once, fine-tune everywhere" recipe defined the post-2018 NLP landscape and set new state-of-the-art on the GLUE, SQuAD, and SWAG benchmarks at publication time.