class

GPT2Model

extendsPretrainedModel

GPT2Model(config: GPT2Config)

source edit

GPT-2 decoder trunk — N pre-LN transformer blocks topped by ln_f.

Implements the modified architecture of Radford, Wu, Child, Luan, Amodei, and Sutskever, 2019. Differs from GPT-1 in three ways: (i) pre-LN blocks — LayerNorm is applied to the residual input of each sub-layer rather than after the residual add, which stabilises optimisation past depth $L \geq 24$ ; (ii) a final LayerNorm (ln_f) caps the trunk; (iii) residual-output projection weights are initialised with an extra $1/\sqrt{2N}$ factor (Radford 2019 §2.3) to keep activation variance roughly invariant to depth.

Parameters

configGPT2Config

Hyperparameters controlling vocabulary, depth, width, head count, and the residual-init scaling toggle.

Attributes

wtenn.Embedding

Word / token embedding table of shape (vocab_size, hidden_size). HuggingFace name retained verbatim for state-dict portability.

wpenn.Embedding

Learned absolute position embedding of shape (max_position_embeddings, hidden_size).

dropnn.Dropout

Post-embedding dropout layer.

hnn.ModuleList

Stack of config.num_hidden_layers pre-LN transformer blocks.

ln_fnn.LayerNorm

Final LayerNorm applied to the trunk output.

config_classtype[GPT2Config]

Registry pointer for matching-config instantiation from disk.

base_model_prefixstr

State-dict prefix ("transformer") under which the trunk is nested in task-head variants.

Notes

Reference: Radford, Wu, Child, Luan, Amodei, and Sutskever, "Language Models are Unsupervised Multitask Learners", OpenAI Technical Report, 2019.

Each pre-LN block computes

\begin{aligned} x &\leftarrow x + \mathrm{Attn}(\mathrm{LN}_1(x)) \\ x &\leftarrow x + \mathrm{MLP}(\mathrm{LN}_2(x)) \end{aligned}

so the residual stream carries un-normalised state. The trunk produces final hidden states $H \in \mathbb{R}^{B \times T \times H}$ which the LM head projects to vocabulary logits.

Examples

>>> import lucid
>>> from lucid.models.text.gpt2 import GPT2Config, GPT2Model
>>> cfg = GPT2Config(num_hidden_layers=2, hidden_size=128,
...                  num_attention_heads=2, intermediate_size=512)
>>> model = GPT2Model(cfg).eval()
>>> input_ids = lucid.tensor([[15496, 11, 995, 13]])   # "Hello, world."
>>> out = model(input_ids)
>>> out.last_hidden_state.shape   # (B=1, T=4, H=128)
(1, 4, 128)

Used by 2

Constructors

dunder

init

→None

__init__(config: GPT2Config)

source edit

Instance methods

forward

→BaseModelOutput

forward(input_ids: Tensor, attention_mask: Tensor | None = None, past_key_values: Cache | None = None, use_cache: bool = False, cache_position: Tensor | None = None)

source edit

get_input_embeddings

→nn.Module

get_input_embeddings()

source edit

set_input_embeddings

→None

set_input_embeddings(value: nn.Module)

source edit

>>> import lucid >>> from lucid.models.text.gpt2 import GPT2Config, GPT2Model >>> cfg = GPT2Config(num_hidden_layers=2, hidden_size=128, ... num_attention_heads=2, intermediate_size=512) >>> model = GPT2Model(cfg).eval() >>> input_ids = lucid.tensor([[15496, 11, 995, 13]]) # "Hello, world." >>> out = model(input_ids) >>> out.last_hidden_state.shape # (B=1, T=4, H=128) (1, 4, 128)