gpt

→GPTModel

gpt(pretrained: bool | str = False, weights: GPTWeights | None = None, overrides: object = {})

source edit

Construct a GPT-1 decoder-only Transformer trunk.

The original generative pre-training transformer from Radford, Narasimhan, Salimans, and Sutskever, 2018: $L=12$ transformer blocks, $H=768$ hidden, $A=12$ attention heads, intermediate width 3072, learned absolute position embeddings, and a 40,478-piece BPE vocabulary trained on BookCorpus. Roughly 117M parameters — the seed of the modern decoder-only LLM family.

Model Size

Parameters

pretrainedbool= False

Reserved for future weight registration. No checkpoints are wired yet, so the model is returned with random initialisation.

weights(GPTWeights, optional, keyword - only)= None

Explicit weights enum member; takes precedence over pretrained.

**overridesobject= {}

Optional GPTConfig field overrides (e.g. vocab_size=..., hidden_size=..., num_hidden_layers=...) forwarded into the underlying config.

Returns

GPTModel

Decoder trunk configured with GPT-1 defaults plus any overrides.

Notes

Reference: Radford, Narasimhan, Salimans, and Sutskever, "Improving Language Understanding by Generative Pre-Training", OpenAI Technical Report, 2018.

Causal language-model factorisation:

\mathcal{L}_{\mathrm{LM}}(\theta) = \sum_{t=1}^{T} \log p_\theta(x_t \mid x_{<t}),

enforced architecturally by a lower-triangular mask inside scaled dot-product attention so position $t$ only attends to $\{1, \dots, t\}$ .

Examples

>>> import lucid
>>> from lucid.models.text.gpt import gpt
>>> model = gpt().eval()
>>> input_ids = lucid.tensor([[101, 7592, 2088, 102]])
>>> out = model(input_ids)
>>> out.last_hidden_state.shape   # (B=1, T=4, H=768)
(1, 4, 768)

Used by 1

lucid.models.text.gpt

>>> import lucid >>> from lucid.models.text.gpt import gpt >>> model = gpt().eval() >>> input_ids = lucid.tensor([[101, 7592, 2088, 102]]) >>> out = model(input_ids) >>> out.last_hidden_state.shape # (B=1, T=4, H=768) (1, 4, 768)