Bare GPT-1 decoder-only Transformer trunk.
Implements the original generative pre-training architecture from
Radford, Narasimhan, Salimans, and Sutskever, 2018: a 12-layer stack
of decoder blocks with fused-QKV multi-head causal self-attention, a
2-layer feed-forward block, post-LayerNorm residuals, learned absolute
position embeddings, and a tanh-approximated GELU activation
(gelu_new). Returns the per-token hidden states; the LM and
classifier heads sit on top of this trunk.
Parameters
configGPTConfigHyperparameters controlling vocabulary, depth, width, head count,
and regularisation. See
GPTConfig for the full field
list.Attributes
tokens_embednn.EmbeddingToken embedding table of shape
(vocab_size, hidden_size).
HuggingFace name retained for state-dict portability.positions_embednn.EmbeddingLearned absolute position embedding of shape
(max_position_embeddings, hidden_size).dropnn.DropoutPost-embedding dropout layer.
Stack of
config.num_hidden_layers GPT-1 transformer blocks.config_classtype[GPTConfig]Registry pointer for matching-config instantiation.
base_model_prefixstrState-dict prefix (
"transformer") under which the trunk is
nested in task-head variants.Notes
Reference: Radford, Narasimhan, Salimans, and Sutskever, "Improving Language Understanding by Generative Pre-Training", OpenAI Technical Report, 2018.
Causal LM factorisation:
Examples
>>> import lucid
>>> from lucid.models.text.gpt import GPTConfig, GPTModel
>>> cfg = GPTConfig(num_hidden_layers=2, hidden_size=128,
... num_attention_heads=2, intermediate_size=512)
>>> model = GPTModel(cfg).eval()
>>> input_ids = lucid.tensor([[101, 7592, 2088, 102]])
>>> out = model(input_ids)
>>> out.last_hidden_state.shape # (B=1, T=4, H=128)
(1, 4, 128)Used by 2
Constructors
1Instance methods
3forward(input_ids: Tensor, attention_mask: Tensor | None = None, past_key_values: Cache | None = None, use_cache: bool = False, cache_position: Tensor | None = None)