class

GPTLMHeadModel

extendsPretrainedModelCausalLMMixin

GPTLMHeadModel(config: GPTConfig)

source edit

GPT-1 with a tied causal-language-modeling head.

Wraps GPTModel with an output linear whose weight matrix is bound to the input tokens_embed table when config.tie_word_embeddings is True — halving the parameter cost of the softmax layer. This is both the pre-training entry point and the host of lucid.models.CausalLMMixin.generate for free-form autoregressive sampling.

Parameters

configGPTConfig

Hyperparameters. config.tie_word_embeddings (default True) controls weight tying.

Attributes

transformerGPTModel

Underlying decoder trunk.

lm_headnn.Linear

Output projection of shape (hidden_size, vocab_size) mapping each token's hidden state to vocabulary logits.

Notes

Reference: Radford et al., "Improving Language Understanding by Generative Pre-Training", OpenAI Technical Report, 2018.

Loss when labels is supplied is the standard next-token-shifted causal-LM cross-entropy

\mathcal{L}_{\mathrm{CLM}} = -\frac{1}{B(T-1)} \sum_{b, t} \log p_\theta(y_{b, t+1} \mid x_{b, \le t}),

with positions labelled -100 excluded.

Examples

>>> import lucid
>>> from lucid.models.text.gpt import GPTConfig, GPTLMHeadModel
>>> cfg = GPTConfig(num_hidden_layers=2, hidden_size=128,
...                 num_attention_heads=2, intermediate_size=512)
>>> model = GPTLMHeadModel(cfg).eval()
>>> input_ids = lucid.tensor([[101, 7592, 2088, 102]])
>>> out = model(input_ids)
>>> out.logits.shape   # (B=1, T=4, V=40478)
(1, 4, 40478)

Used by 2

Constructors

dunder

init

→None

__init__(config: GPTConfig)

source edit

Instance methods

forward

→CausalLMOutput

forward(input_ids: Tensor, attention_mask: Tensor | None = None, labels: Tensor | None = None, past_key_values: Cache | None = None, use_cache: bool = False, cache_position: Tensor | None = None)

source edit

>>> import lucid >>> from lucid.models.text.gpt import GPTConfig, GPTLMHeadModel >>> cfg = GPTConfig(num_hidden_layers=2, hidden_size=128, ... num_attention_heads=2, intermediate_size=512) >>> model = GPTLMHeadModel(cfg).eval() >>> input_ids = lucid.tensor([[101, 7592, 2088, 102]]) >>> out = model(input_ids) >>> out.logits.shape # (B=1, T=4, V=40478) (1, 4, 40478)