gpt_lm

→GPTLMHeadModel

gpt_lm(pretrained: bool | str = False, weights: GPTLMWeights | None = None, overrides: object = {})

source edit

Construct a GPT-1 model with the tied causal-LM head.

Same trunk as gpt (L=12, H=768, A=12, ~117M parameters), wrapped with a tied LM head that reuses the input token-embedding matrix as the output projection. This is the pre-training entry point and the host of lucid.models.CausalLMMixin.generate for free-form autoregressive sampling.

Model Size

Parameters

pretrainedbool= False

Reserved for future weight registration; currently a no-op.

weights(GPTLMWeights, optional, keyword - only)= None

Explicit weights enum member; takes precedence over pretrained.

**overridesobject= {}

Optional GPTConfig field overrides forwarded into the underlying config.

Returns

GPTLMHeadModel

GPT-1 trunk wrapped with the tied LM head.

Notes

Reference: Radford et al., "Improving Language Understanding by Generative Pre-Training", OpenAI Technical Report, 2018.

When labels is supplied the head computes the standard next-token-shifted causal-LM loss

\mathcal{L}_{\mathrm{CLM}} = -\frac{1}{B(T-1)} \sum_{b,t} \log p_\theta(y_{b, t+1} \mid x_{b, \le t}),

with positions labelled -100 excluded.

Examples

>>> import lucid
>>> from lucid.models.text.gpt import gpt_lm
>>> model = gpt_lm().eval()
>>> input_ids = lucid.tensor([[101, 7592, 2088, 102]])
>>> out = model(input_ids)
>>> out.logits.shape   # (B=1, T=4, V=40478)
(1, 4, 40478)

Used by 1

lucid.models.text.gpt

>>> import lucid >>> from lucid.models.text.gpt import gpt_lm >>> model = gpt_lm().eval() >>> input_ids = lucid.tensor([[101, 7592, 2088, 102]]) >>> out = model(input_ids) >>> out.logits.shape # (B=1, T=4, V=40478) (1, 4, 40478)