class

GPT2LMHeadModel

extendsPretrainedModelCausalLMMixin

GPT2LMHeadModel(config: GPT2Config)

source edit

GPT-2 with a tied causal-language-modeling head.

Wraps GPT2Model with an output linear projection whose weight matrix is bound to the input wte embedding table when config.tie_word_embeddings is True — halving the parameter cost of the softmax layer. This is the entry point for lucid.models.CausalLMMixin.generate and is the standard recipe for both pre-training and downstream generative use.

Parameters

configGPT2Config

Hyperparameters. config.tie_word_embeddings (default True) controls whether the LM-head weight is shared with the input embedding matrix.

Attributes

transformerGPT2Model

Underlying decoder trunk (HF naming: transformer.*).

lm_headnn.Linear

Output projection of shape (hidden_size, vocab_size) mapping each token's hidden state to vocabulary logits.

Notes

Reference: Radford, Wu, Child, Luan, Amodei, and Sutskever, "Language Models are Unsupervised Multitask Learners", OpenAI Technical Report, 2019.

The training loss (when labels is supplied) is the next-token shifted causal-LM cross-entropy:

\mathcal{L}_{\mathrm{CLM}} = -\frac{1}{B(T-1)} \sum_{b, t} \log p_\theta(y_{b, t+1} \mid x_{b, \le t}),

with positions labelled -100 excluded.

Examples

>>> import lucid
>>> from lucid.models.text.gpt2 import GPT2Config, GPT2LMHeadModel
>>> cfg = GPT2Config(num_hidden_layers=2, hidden_size=128,
...                  num_attention_heads=2, intermediate_size=512)
>>> model = GPT2LMHeadModel(cfg).eval()
>>> input_ids = lucid.tensor([[15496, 11, 995, 13]])
>>> out = model(input_ids)
>>> out.logits.shape   # (B=1, T=4, V=50257)
(1, 4, 50257)

Used by 2

Constructors

dunder

init

→None

__init__(config: GPT2Config)

source edit

Instance methods

forward

→CausalLMOutput

forward(input_ids: Tensor, attention_mask: Tensor | None = None, labels: Tensor | None = None, past_key_values: Cache | None = None, use_cache: bool = False, cache_position: Tensor | None = None)

source edit

>>> import lucid >>> from lucid.models.text.gpt2 import GPT2Config, GPT2LMHeadModel >>> cfg = GPT2Config(num_hidden_layers=2, hidden_size=128, ... num_attention_heads=2, intermediate_size=512) >>> model = GPT2LMHeadModel(cfg).eval() >>> input_ids = lucid.tensor([[15496, 11, 995, 13]]) >>> out = model(input_ids) >>> out.logits.shape # (B=1, T=4, V=50257) (1, 4, 50257)